Jul 24 12:52:55.899216 ip-10-0-141-29 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 12:52:56.289070 ip-10-0-141-29 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:52:56.289070 ip-10-0-141-29 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 12:52:56.289070 ip-10-0-141-29 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:52:56.289070 ip-10-0-141-29 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 12:52:56.289070 ip-10-0-141-29 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:52:56.289793 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.289134    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 12:52:56.295530 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295508    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:52:56.295530 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295526    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:52:56.295530 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295530    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:52:56.295530 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295533    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295538    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295543    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295546    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295550    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295553    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295556    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295558    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295561    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295564    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295566    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295569    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295579    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295582    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295585    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295587    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295589    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295592    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295595    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295597    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:52:56.295641 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295600    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295603    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295605    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295608    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295610    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295613    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295616    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295620    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295623    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295627    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295629    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295633    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295635    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295638    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295641    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295643    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295646    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295649    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295652    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:52:56.296132 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295654    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295657    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295660    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295662    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295666    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295668    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295671    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295673    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295676    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295678    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295681    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295683    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295686    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295688    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295691    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295694    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295696    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295699    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295701    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:52:56.296598 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295704    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295709    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295712    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295715    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295717    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295720    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295723    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295726    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295730    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295733    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295736    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295738    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295741    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295745    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295748    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295751    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295754    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295757    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295760    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295763    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:52:56.297055 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295765    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295768    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295770    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295773    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295775    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295778    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295780    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295782    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295786    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295788    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295791    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295793    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295796    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295799    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295801    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295804    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295806    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.295809    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296515    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296522    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:52:56.297518 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296525    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296529    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296532    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296535    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296538    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296540    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296543    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296545    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296548    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296550    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296553    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296556    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296559    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296562    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296564    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296567    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296569    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296572    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296574    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:52:56.298038 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296577    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296580    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296584    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296586    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296589    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296591    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296594    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296598    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296602    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296606    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296609    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296612    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296615    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296618    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296620    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296623    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296625    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296629    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296632    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296634    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:52:56.298520 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296637    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296639    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296642    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296645    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296647    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296650    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296653    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296656    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296658    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296661    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296664    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296666    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296669    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296671    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296674    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296676    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296680    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296683    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296685    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296688    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:52:56.299022 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296691    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296693    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296695    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296699    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296702    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296705    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296707    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296710    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296713    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296715    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296717    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296720    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296722    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296724    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296727    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296730    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296732    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296735    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296737    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296740    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:52:56.299483 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296742    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296745    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296747    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296750    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296752    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296755    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296757    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296761    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296764    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296767    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296770    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296773    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296775    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296778    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296781    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296784    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296786    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.296789    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297172    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:52:56.299973 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297177    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297180    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297183    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297185    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297188    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297190    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297193    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297196    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297198    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297201    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297203    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297206    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297209    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297212    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297214    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297217    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297219    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297222    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297225    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:52:56.300427 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297228    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297230    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297233    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297235    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297238    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297240    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297242    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297245    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297249    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297253    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297256    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297259    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297262    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297265    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297268    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297271    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297275    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297278    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297280    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297283    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:52:56.300871 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297285    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297287    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297290    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297292    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297295    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297298    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297300    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297303    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297305    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297307    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297311    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297315    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297320    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297323    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297325    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297328    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297331    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297333    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297336    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:52:56.301350 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297339    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297342    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297344    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297347    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297349    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297352    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297354    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297357    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297359    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297362    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297364    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297367    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297369    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297372    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297374    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297377    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297379    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297382    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297384    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297387    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:52:56.301803 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297390    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297392    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297395    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297397    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297401    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297405    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297408    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297410    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297413    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297416    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297418    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297421    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297424    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297426    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297428    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297431    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297433    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297436    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297439    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.297443    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:52:56.302289 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297514    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297525    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297534    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297540    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297549    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297554    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297559    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297563    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297566    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297569    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297572    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297575    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297579    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297582    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297585    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297588    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297592    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297597    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297601    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297604    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297607    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297610    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297613    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297617    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 12:52:56.302761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297619    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297622    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297626    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297629    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297632    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297635    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297638    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297640    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297644    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297647    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297650    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297652    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297656    2579 flags.go:64] FLAG: --enable-server="true"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297659    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297663    2579 flags.go:64] FLAG: --event-burst="100"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297666    2579 flags.go:64] FLAG: --event-qps="50"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297668    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297671    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297674    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297677    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297680    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297683    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297686    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297689    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297693    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 12:52:56.303326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297698    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297701    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297703    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297706    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297709    2579 flags.go:64] FLAG: --feature-gates=""
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297713    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297716    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297719    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297722    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297725    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297728    2579 flags.go:64] FLAG: --help="false"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297730    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297733    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297736    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297739    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297742    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297745    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297748    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297751    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297753    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297756    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297759    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297762    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297764    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 24 12:52:56.303993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297767    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297769    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297772    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297775    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297778    2579 flags.go:64] FLAG: --lock-file=""
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297781    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297783    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297787    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297793    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297796    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297799    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297801    2579 flags.go:64] FLAG: --logging-format="text"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297804    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297807    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297810    2579 flags.go:64] FLAG: --manifest-url=""
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297812    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297816    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297819    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297823    2579 flags.go:64] FLAG: --max-pods="110"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297826    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297829    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297831    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297834    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297837    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297840    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 12:52:56.304596 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297843    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297850    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297853    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297856    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297858    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297861    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297866    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297869    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297872    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297875    2579 flags.go:64] FLAG: --port="10250"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297878    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297881    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-022a80f864d50c811"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297883    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297887    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297891    2579 flags.go:64] FLAG: --register-node="true"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297893    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297898    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297901    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297903    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297906    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297910    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297912    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297915    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297918    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297935    2579 flags.go:64] FLAG: --runonce="false"
Jul 24 12:52:56.305623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297937    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297941    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297944    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297947    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297950    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297953    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297956    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297959    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297962    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297964    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297967    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297970    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297972    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297975    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297978    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297983    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297986    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297988    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297993    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297996    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.297999    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.298004    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.298007    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.298010    2579 flags.go:64] FLAG: --v="2"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.298014    2579 flags.go:64] FLAG: --version="false"
Jul 24 12:52:56.306530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.298018    2579 flags.go:64] FLAG: --vmodule=""
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.298021    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.298024    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298159    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298164    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298168    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298173    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298176    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298179    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298182    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298184    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298187    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298189    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298192    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298194    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298198    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298200    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298202    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298205    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:52:56.307149 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298207    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298210    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298212    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298215    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298217    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298219    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298222    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298224    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298227    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298230    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298233    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298237    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298240    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298243    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298246    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298248    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298250    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298253    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298255    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298258    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:52:56.307591 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298260    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298263    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298266    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298268    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298271    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298273    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298275    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298278    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298281    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298283    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298286    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298288    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298291    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298293    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298296    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298298    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298300    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298303    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298305    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298308    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:52:56.308202 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298310    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298313    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298316    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298320    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298323    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298326    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298328    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298331    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298333    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298336    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298339    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298341    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298344    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298346    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298349    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298352    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298354    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298357    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298359    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:52:56.308678 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298361    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298364    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298367    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298369    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298372    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298374    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298376    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298380    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298383    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298386    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298389    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298391    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298394    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298397    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298399    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298402    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298406    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298409    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298411    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298414    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:52:56.309154 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298416    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298419    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298421    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.298424    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.299029    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.307504    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.307520    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.307542    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.307551    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307612    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307617    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307620    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307623    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307627    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:52:56.309619 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307629    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307634    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307638    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307641    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307644    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307647    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307649    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307652    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307655    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307658    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307660    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307663    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307666    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307668    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307671    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307674    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307678    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307680    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307683    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:52:56.309985 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307685    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307688    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307691    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307693    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307696    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307699    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307701    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307703    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307706    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307708    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307711    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307713    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307717    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307719    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307722    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307725    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307727    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307730    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307732    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:52:56.310431 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307735    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307737    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307740    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307742    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307745    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307747    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307750    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307753    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307755    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307758    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307761    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307763    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307766    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307768    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307771    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307773    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307776    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307778    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307781    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307783    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:52:56.310876 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307786    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307788    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307791    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307793    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307796    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307798    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307801    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307803    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307806    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307808    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307811    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307814    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307816    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307818    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307821    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307823    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307826    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307829    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307831    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307834    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:52:56.311374 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307836    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307840    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307844    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307847    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307850    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307852    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307855    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307857    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307860    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307863    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307866    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307869    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307872    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307875    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307878    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:52:56.311877 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.307881    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.307887    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308054    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308061    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308065    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308068    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308071    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308073    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308076    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308079    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308081    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308084    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308086    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308089    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:52:56.312259 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308092    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308095    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308097    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308100    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308102    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308105    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308107    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308110    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308112    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308115    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308117    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308120    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308122    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308125    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308127    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308130    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308132    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308135    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308137    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308140    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:52:56.312644 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308142    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308145    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308148    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308150    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308152    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308155    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308158    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308160    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308163    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308165    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308168    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308170    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308173    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308176    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308179    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308181    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308183    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308187    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308191    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308194    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:52:56.313196 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308196    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308199    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308201    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308204    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308206    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308209    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308212    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308214    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308217    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308219    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308222    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308224    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308227    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308230    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308232    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308235    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308237    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308240    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308243    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308245    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:52:56.313671 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308247    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308250    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308253    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308256    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308258    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308261    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308266    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308269    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308272    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308275    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308278    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308280    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308283    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308285    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308288    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308290    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308293    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308295    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308297    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:52:56.314152 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308301    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308304    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308307    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308310    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308312    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308316    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308318    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:56.308321    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.308325    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.308445    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.311147    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.312730    2579 server.go:1019] "Starting client certificate rotation"
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.312827    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 12:52:56.314603 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.312859    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 12:52:56.315323 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.315310    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 12:52:56.316454 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.316426    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 12:52:56.324243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.324225    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 24 12:52:56.326162 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.326147    2579 log.go:25] "Validated CRI v1 image API"
Jul 24 12:52:56.327093 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.327079    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 12:52:56.332569 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.332543    2579 fs.go:135] Filesystem UUIDs: map[5ace2507-41c6-4dd0-8cef-7012dc3d5be3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9b03eb93-729b-45fc-b061-ac7c2c445310:/dev/nvme0n1p4]
Jul 24 12:52:56.332641 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.332569    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 12:52:56.341360 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.341239    2579 manager.go:216] Machine: {Timestamp:2026-07-24 12:52:56.33953968 +0000 UTC m=+0.328858367 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3190444 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca83e13dbed03f92ed6ef79e52373 SystemUUID:ec2ca83e-13db-ed03-f92e-d6ef79e52373 BootID:aefd056b-1044-43fa-844f-595d4ca7f26c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f7:51:56:7b:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f7:51:56:7b:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:0d:e3:05:d4:03 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 12:52:56.341360 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.341348    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 12:52:56.341502 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.341435    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 12:52:56.344452 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.344425    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 12:52:56.344578 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.344454    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 12:52:56.345744 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.345731    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 12:52:56.345744 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.345745    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 12:52:56.345854 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.345753    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 12:52:56.346361 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.346351    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 12:52:56.347450 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.347440    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 12:52:56.348543 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.348528    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 12:52:56.350189 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350178    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 12:52:56.350300 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350291    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 12:52:56.350328 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350321    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 12:52:56.350354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350331    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 12:52:56.350354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350345    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 12:52:56.350411 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350359    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 24 12:52:56.350411 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350367    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 12:52:56.350733 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350721    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 12:52:56.350784 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.350748    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 12:52:56.353666 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.353649    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 12:52:56.354887 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.354874    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 12:52:56.354957 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.354903    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 12:52:56.355489 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355479    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 12:52:56.355518 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355500    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 12:52:56.355518 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355508    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 12:52:56.355518 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355517    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 12:52:56.355618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355528    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 12:52:56.355618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355537    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 12:52:56.355618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355585    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 12:52:56.355618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355592    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 12:52:56.355618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355599    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 12:52:56.355618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355604    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 12:52:56.355618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355612    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 12:52:56.355618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.355620    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 12:52:56.356222 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.356213    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 12:52:56.356222 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.356222    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 12:52:56.359215 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.359195    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 12:52:56.359459 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.359448    2579 server.go:1268] "Started kubelet"
Jul 24 12:52:56.359548 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.359519    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 12:52:56.359757 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.359687    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 12:52:56.359829 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.359784    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 12:52:56.360069 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.360043    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 12:52:56.360138 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.360042    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 12:52:56.360335 ip-10-0-141-29 systemd[1]: Started Kubernetes Kubelet.
Jul 24 12:52:56.361446 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.361415    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 12:52:56.361788 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.361774    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 12:52:56.364937 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.364903    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 12:52:56.366939 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.366912    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 12:52:56.366939 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.366919    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 12:52:56.367146 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.367123    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:56.367580 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.367565    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 12:52:56.367580 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.367580    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 12:52:56.367762 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.367748    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 12:52:56.367762 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.367762    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 12:52:56.367849 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.367797    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 12:52:56.367849 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.367828    2579 factory.go:55] Registering systemd factory
Jul 24 12:52:56.367849 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.367843    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 24 12:52:56.368191 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.368175    2579 factory.go:153] Registering CRI-O factory
Jul 24 12:52:56.368254 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.368193    2579 factory.go:223] Registration of the crio container factory successfully
Jul 24 12:52:56.368254 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.368248    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 12:52:56.368316 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.368271    2579 factory.go:103] Registering Raw factory
Jul 24 12:52:56.368316 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.368285    2579 manager.go:1196] Started watching for new ooms in manager
Jul 24 12:52:56.368651 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.368630    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qp2hz"
Jul 24 12:52:56.369034 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.369018    2579 manager.go:349] Starting recovery of all containers
Jul 24 12:52:56.376034 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.368640    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.ec2.internal.18c53ae38f5bdbcf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.ec2.internal,UID:ip-10-0-141-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.ec2.internal,},FirstTimestamp:2026-07-24 12:52:56.359418831 +0000 UTC m=+0.348737519,LastTimestamp:2026-07-24 12:52:56.359418831 +0000 UTC m=+0.348737519,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.ec2.internal,}"
Jul 24 12:52:56.376034 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.371049    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 12:52:56.376034 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.371089    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 12:52:56.377941 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.377901    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qp2hz"
Jul 24 12:52:56.380676 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.380530    2579 manager.go:354] Recovery completed
Jul 24 12:52:56.385372 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.385358    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:52:56.387131 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.387116    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:52:56.387198 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.387146    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:52:56.387198 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.387156    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:52:56.387718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.387704    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 12:52:56.387763 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.387719    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 12:52:56.387763 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.387739    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 12:52:56.387763 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.387709    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:52:56.388459 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.388445    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:52:56.388521 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.388470    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:52:56.388521 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.388482    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:52:56.388957 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.388937    2579 policy_none.go:49] "None policy: Start"
Jul 24 12:52:56.388957 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.388961    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 12:52:56.389032 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.388972    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 12:52:56.389457 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.389391    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-29.ec2.internal.18c53ae39102bb4d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-29.ec2.internal,UID:ip-10-0-141-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-29.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-29.ec2.internal,},FirstTimestamp:2026-07-24 12:52:56.387132237 +0000 UTC m=+0.376450925,LastTimestamp:2026-07-24 12:52:56.387132237 +0000 UTC m=+0.376450925,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-29.ec2.internal,}"
Jul 24 12:52:56.389887 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.389877    2579 policy_none.go:47] "Start"
Jul 24 12:52:56.424514 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.424496    2579 manager.go:337] "Starting Device Plugin manager"
Jul 24 12:52:56.424606 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.424537    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 12:52:56.424606 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.424548    2579 server.go:87] "Starting device plugin registration server"
Jul 24 12:52:56.424822 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.424806    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 12:52:56.424944 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.424821    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 12:52:56.424944 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.424937    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 12:52:56.425048 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.425016    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 12:52:56.425048 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.425023    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 12:52:56.425632 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.425615    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 12:52:56.425712 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.425668    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:56.525183 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.525151    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:52:56.528877 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.528856    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:52:56.529001 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.528887    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:52:56.529001 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.528897    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:52:56.529001 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.528947    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.537836 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.537817    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.537887 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.537845    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-29.ec2.internal\": node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:56.555139 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.555052    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 12:52:56.556480 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.556463    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 12:52:56.556537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.556511    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 12:52:56.556584 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.556541    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 12:52:56.556639 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.556577    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 12:52:56.558991 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.558967    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:52:56.560701 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.560686    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:56.657077 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.657042    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"]
Jul 24 12:52:56.657187 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.657137    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:52:56.658161 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.658144    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:52:56.658235 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.658176    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:52:56.658235 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.658188    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:52:56.659356 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.659343    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:52:56.659496 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.659482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.659554 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.659515    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:52:56.660147 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.660129    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:52:56.660230 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.660148    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:52:56.660230 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.660163    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:52:56.660230 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.660168    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:52:56.660230 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.660174    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:52:56.660230 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.660187    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:52:56.661282 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.661260    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:56.661332 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.661314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.661365 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.661334    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:52:56.662031 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.662015    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:52:56.662114 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.662043    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:52:56.662114 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.662081    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:52:56.669613 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.669596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61a709d601992dbacaba1dc40727bd91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"61a709d601992dbacaba1dc40727bd91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.669700 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.669620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a709d601992dbacaba1dc40727bd91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"61a709d601992dbacaba1dc40727bd91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.669700 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.669645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/630256be83544d972c8b0ff554f26b54-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.ec2.internal\" (UID: \"630256be83544d972c8b0ff554f26b54\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.685745 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.685724    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-29.ec2.internal\" not found" node="ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.690086 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.690072    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-29.ec2.internal\" not found" node="ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.761567 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.761540    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:56.769949 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.769914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61a709d601992dbacaba1dc40727bd91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"61a709d601992dbacaba1dc40727bd91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.769998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.769961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a709d601992dbacaba1dc40727bd91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"61a709d601992dbacaba1dc40727bd91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.770030 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.769998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/630256be83544d972c8b0ff554f26b54-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.ec2.internal\" (UID: \"630256be83544d972c8b0ff554f26b54\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.770030 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.770009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61a709d601992dbacaba1dc40727bd91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"61a709d601992dbacaba1dc40727bd91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.770104 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.770080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/630256be83544d972c8b0ff554f26b54-config\") pod \"kube-apiserver-proxy-ip-10-0-141-29.ec2.internal\" (UID: \"630256be83544d972c8b0ff554f26b54\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.770104 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.770095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a709d601992dbacaba1dc40727bd91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal\" (UID: \"61a709d601992dbacaba1dc40727bd91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.862523 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.862446    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:56.963076 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:56.963035    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:56.987563 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.987527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:56.992786 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:56.992766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:57.063741 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.063692    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:57.164298 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.164233    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:57.264841 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.264795    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-29.ec2.internal\" not found"
Jul 24 12:52:57.312416 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.312391    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 12:52:57.312884 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.312522    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 12:52:57.330660 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.330636    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:52:57.350659 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.350633    2579 apiserver.go:52] "Watching apiserver"
Jul 24 12:52:57.362427 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.362399    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 12:52:57.362739 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.362718    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-ldqh9","openshift-network-operator/iptables-alerter-v5kqw","kube-system/global-pull-secret-syncer-shckb","openshift-multus/network-metrics-daemon-8nqcd","openshift-network-diagnostics/network-check-target-nn8v9","openshift-ovn-kubernetes/ovnkube-node-x2pkb","kube-system/konnectivity-agent-stggw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4","openshift-cluster-node-tuning-operator/tuned-tktw2","openshift-dns/node-resolver-qf5nm","openshift-image-registry/node-ca-zl565","openshift-multus/multus-72tx8"]
Jul 24 12:52:57.364249 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.364230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.365426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.365409    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.366862 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.366838    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x7gvx\""
Jul 24 12:52:57.367021 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.366957    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 12:52:57.367021 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.366992    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 12:52:57.367021 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.367006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.367182 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.367144    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:52:57.367424 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.367411    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:57.367857 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.367844    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 12:52:57.368158 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.368144    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 12:52:57.368257 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.368242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 12:52:57.368310 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.368302    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bjxlp\""
Jul 24 12:52:57.368733 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.368719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.369793 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.369774    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 12:52:57.369891 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.369803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 12:52:57.369891 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.369818    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 12:52:57.369891 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.369842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.369891 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.369778    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 12:52:57.370252 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.370238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6mrxj\""
Jul 24 12:52:57.371096 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.371080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.371164 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.371145    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:52:57.371527 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.371513    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gqmxb\""
Jul 24 12:52:57.371565 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.371528    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 12:52:57.371565 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.371514    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 12:52:57.372454 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.372440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:57.372527 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.372507    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:52:57.372964 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.372948    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 12:52:57.373031 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.372992    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:52:57.373081 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.372992    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vjqbq\""
Jul 24 12:52:57.373128 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373043    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 12:52:57.373176 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-systemd\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373176 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-run\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373244 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-cni-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373244 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-os-release\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373244 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-k8s-cni-cncf-io\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkqj\" (UniqueName: \"kubernetes.io/projected/a0e2ce4c-941e-4532-9ef8-27420e18a33b-kube-api-access-cvkqj\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-host\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnx89\" (UniqueName: \"kubernetes.io/projected/82751748-2f8d-49e3-b9ae-3769ec465016-kube-api-access-xnx89\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-kubelet\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373531 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-hostroot\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373531 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-etc-kubernetes\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373531 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysconfig\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373531 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-lib-modules\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373531 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82751748-2f8d-49e3-b9ae-3769ec465016-tmp\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373681 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-cni-multus\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373681 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:52:57.373681 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.373610    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:52:57.373681 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysctl-d\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373681 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-sys\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373681 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82751748-2f8d-49e3-b9ae-3769ec465016-etc-tuned\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373681 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a39d5644-41ba-439f-87e2-72f2dfae3422-serviceca\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.373681 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0e2ce4c-941e-4532-9ef8-27420e18a33b-cni-binary-copy\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a39d5644-41ba-439f-87e2-72f2dfae3422-host\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnkg\" (UniqueName: \"kubernetes.io/projected/a39d5644-41ba-439f-87e2-72f2dfae3422-kube-api-access-cpnkg\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-modprobe-d\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-var-lib-kubelet\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-cnibin\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-conf-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-daemon-config\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.373890 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-socket-dir-parent\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-netns\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-cni-bin\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-multus-certs\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.373999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-os-release\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.374023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-kubernetes\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.374037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-system-cni-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.374056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cnibin\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.374078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.374100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc9jt\" (UniqueName: \"kubernetes.io/projected/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-kube-api-access-rc9jt\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.374251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.374115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysctl-conf\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.374739 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.374724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.375912 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.375897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:52:57.377690 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.377676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.378011 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.377992    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 12:52:57.378096 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.378012    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 12:52:57.378096 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.377999    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jq85z\""
Jul 24 12:52:57.378186 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.378118    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 12:52:57.378236 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.378193    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 12:52:57.378236 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.378206    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 12:52:57.378330 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.378210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 12:52:57.379079 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.378896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 12:52:57.379079 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.378903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4bm2h\""
Jul 24 12:52:57.379079 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.378964    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 12:52:57.379324 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.379309    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 12:52:57.379406 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.379346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.379406 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.379403    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"
Jul 24 12:52:57.380945 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.380271    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal"]
Jul 24 12:52:57.380945 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.380453    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 12:52:57.380945 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.380688    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 12:52:57.380945 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.380674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-27ddb\""
Jul 24 12:52:57.381162 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.381090    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 12:52:57.381943 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.381643    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 12:47:56 +0000 UTC" deadline="2028-02-27 04:55:01.549868617 +0000 UTC"
Jul 24 12:52:57.381943 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.381712    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13984h2m4.16816104s"
Jul 24 12:52:57.382125 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.382109    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 12:52:57.382599 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.382583    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zv69f\""
Jul 24 12:52:57.382761 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.382742    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 12:52:57.382823 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.382749    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 12:52:57.391412 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.391394    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal"]
Jul 24 12:52:57.391599 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.391585    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 12:52:57.405919 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.405902    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m7qpt"
Jul 24 12:52:57.412133 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.412117    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m7qpt"
Jul 24 12:52:57.416622 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.416576    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:52:57.469491 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.469470    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 12:52:57.474430 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.474559 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysconfig\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.474559 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-lib-modules\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.474559 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-cni-multus\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.474559 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-cni-bin\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.474559 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.474559 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-cni-multus\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.474814 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysconfig\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.474814 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b044814-4a86-46db-a651-45578687a328-ovn-node-metrics-cert\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.474814 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-lib-modules\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.474814 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3888e8a-388c-496d-bdbc-8f266fb32067-agent-certs\") pod \"konnectivity-agent-stggw\" (UID: \"b3888e8a-388c-496d-bdbc-8f266fb32067\") " pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:52:57.474814 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.474814 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9v6g\" (UniqueName: \"kubernetes.io/projected/c9bd0504-dd4b-41e5-859a-bf8b54b570af-kube-api-access-h9v6g\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.474814 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7l7t\" (UniqueName: \"kubernetes.io/projected/1b044814-4a86-46db-a651-45578687a328-kube-api-access-j7l7t\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.474814 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-socket-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-sys-fs\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56249eea-b34a-4073-968c-a9e93e64e999-tmp-dir\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a39d5644-41ba-439f-87e2-72f2dfae3422-host\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-ovn\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a39d5644-41ba-439f-87e2-72f2dfae3422-host\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.474978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-node-log\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-registration-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-cnibin\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-conf-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-daemon-config\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-cnibin\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:52:57.475146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-conf-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9bd0504-dd4b-41e5-859a-bf8b54b570af-iptables-alerter-script\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-cni-netd\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6956353c-c03e-4a3c-8a33-a098f4c11e35-dbus\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-kubernetes\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-system-cni-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cnibin\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-kubernetes\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5pw\" (UniqueName: \"kubernetes.io/projected/2e1bc47e-ead0-460e-92ba-e57dfafcde04-kube-api-access-sh5pw\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-system-cni-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-run-ovn-kubernetes\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cnibin\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-device-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-systemd\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysctl-conf\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.475794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-cni-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-k8s-cni-cncf-io\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-log-socket\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysctl-conf\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-kubelet\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-hostroot\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-k8s-cni-cncf-io\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-daemon-config\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-etc-kubernetes\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-kubelet\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9bd0504-dd4b-41e5-859a-bf8b54b570af-host-slash\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-cni-dir\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-hostroot\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-slash\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-etc-kubernetes\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-ovnkube-config\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82751748-2f8d-49e3-b9ae-3769ec465016-tmp\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.476537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-run-netns\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6956353c-c03e-4a3c-8a33-a098f4c11e35-kubelet-config\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56249eea-b34a-4073-968c-a9e93e64e999-hosts-file\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysctl-d\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-sys\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82751748-2f8d-49e3-b9ae-3769ec465016-etc-tuned\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a39d5644-41ba-439f-87e2-72f2dfae3422-serviceca\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-sys\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0e2ce4c-941e-4532-9ef8-27420e18a33b-cni-binary-copy\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-sysctl-d\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.475982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnkg\" (UniqueName: \"kubernetes.io/projected/a39d5644-41ba-439f-87e2-72f2dfae3422-kube-api-access-cpnkg\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-ovnkube-script-lib\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476065    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vcj6\" (UniqueName: \"kubernetes.io/projected/56249eea-b34a-4073-968c-a9e93e64e999-kube-api-access-9vcj6\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-modprobe-d\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.477426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-var-lib-kubelet\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4xx\" (UniqueName: \"kubernetes.io/projected/bca3aa58-22c0-419a-8398-3b1a67b55224-kube-api-access-xc4xx\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-socket-dir-parent\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-netns\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-cni-bin\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-multus-certs\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a39d5644-41ba-439f-87e2-72f2dfae3422-serviceca\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-var-lib-kubelet\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-modprobe-d\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-netns\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0e2ce4c-941e-4532-9ef8-27420e18a33b-cni-binary-copy\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-os-release\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-multus-socket-dir-parent\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.478280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-var-lib-cni-bin\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-host-run-multus-certs\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-etc-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-systemd-units\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-var-lib-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-os-release\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.476985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc9jt\" (UniqueName: \"kubernetes.io/projected/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-kube-api-access-rc9jt\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-kubelet\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-env-overrides\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-systemd\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-run\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-run\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-etc-systemd\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-os-release\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkqj\" (UniqueName: \"kubernetes.io/projected/a0e2ce4c-941e-4532-9ef8-27420e18a33b-kube-api-access-cvkqj\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.479148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3888e8a-388c-496d-bdbc-8f266fb32067-konnectivity-ca\") pod \"konnectivity-agent-stggw\" (UID: \"b3888e8a-388c-496d-bdbc-8f266fb32067\") " pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:52:57.479725 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-host\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.479725 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnx89\" (UniqueName: \"kubernetes.io/projected/82751748-2f8d-49e3-b9ae-3769ec465016-kube-api-access-xnx89\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.479725 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.479725 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0e2ce4c-941e-4532-9ef8-27420e18a33b-os-release\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.479725 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.477668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82751748-2f8d-49e3-b9ae-3769ec465016-host\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.479725 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.479460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82751748-2f8d-49e3-b9ae-3769ec465016-tmp\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.479725 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.479515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82751748-2f8d-49e3-b9ae-3769ec465016-etc-tuned\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.492902 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.492875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnx89\" (UniqueName: \"kubernetes.io/projected/82751748-2f8d-49e3-b9ae-3769ec465016-kube-api-access-xnx89\") pod \"tuned-tktw2\" (UID: \"82751748-2f8d-49e3-b9ae-3769ec465016\") " pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.493026 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.492875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkqj\" (UniqueName: \"kubernetes.io/projected/a0e2ce4c-941e-4532-9ef8-27420e18a33b-kube-api-access-cvkqj\") pod \"multus-72tx8\" (UID: \"a0e2ce4c-941e-4532-9ef8-27420e18a33b\") " pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.500455 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.500436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc9jt\" (UniqueName: \"kubernetes.io/projected/8bb5fd7a-722e-4d5a-98da-135ee39ff1a3-kube-api-access-rc9jt\") pod \"multus-additional-cni-plugins-ldqh9\" (UID: \"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3\") " pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.500850 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.500831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnkg\" (UniqueName: \"kubernetes.io/projected/a39d5644-41ba-439f-87e2-72f2dfae3422-kube-api-access-cpnkg\") pod \"node-ca-zl565\" (UID: \"a39d5644-41ba-439f-87e2-72f2dfae3422\") " pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.525481 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.525452    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a709d601992dbacaba1dc40727bd91.slice/crio-ecbd118f07fcef3dd308d154cae36b151e5e7c656d3f8b8fb1ed76b3b362a130 WatchSource:0}: Error finding container ecbd118f07fcef3dd308d154cae36b151e5e7c656d3f8b8fb1ed76b3b362a130: Status 404 returned error can't find the container with id ecbd118f07fcef3dd308d154cae36b151e5e7c656d3f8b8fb1ed76b3b362a130
Jul 24 12:52:57.532655 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.530703    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:52:57.560443 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.560395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal" event={"ID":"61a709d601992dbacaba1dc40727bd91","Type":"ContainerStarted","Data":"ecbd118f07fcef3dd308d154cae36b151e5e7c656d3f8b8fb1ed76b3b362a130"}
Jul 24 12:52:57.578744 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-ovnkube-script-lib\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578744 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vcj6\" (UniqueName: \"kubernetes.io/projected/56249eea-b34a-4073-968c-a9e93e64e999-kube-api-access-9vcj6\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4xx\" (UniqueName: \"kubernetes.io/projected/bca3aa58-22c0-419a-8398-3b1a67b55224-kube-api-access-xc4xx\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-etc-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-systemd-units\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-var-lib-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-kubelet\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-kubelet\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-etc-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-systemd-units\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-var-lib-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.578953    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:52:57.578998 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.578984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-env-overrides\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3888e8a-388c-496d-bdbc-8f266fb32067-konnectivity-ca\") pod \"konnectivity-agent-stggw\" (UID: \"b3888e8a-388c-496d-bdbc-8f266fb32067\") " pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.579088    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs podName:2e1bc47e-ead0-460e-92ba-e57dfafcde04 nodeName:}" failed. No retries permitted until 2026-07-24 12:52:58.07904075 +0000 UTC m=+2.068359448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs") pod "network-metrics-daemon-8nqcd" (UID: "2e1bc47e-ead0-460e-92ba-e57dfafcde04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-cni-bin\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-cni-bin\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b044814-4a86-46db-a651-45578687a328-ovn-node-metrics-cert\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3888e8a-388c-496d-bdbc-8f266fb32067-agent-certs\") pod \"konnectivity-agent-stggw\" (UID: \"b3888e8a-388c-496d-bdbc-8f266fb32067\") " pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9v6g\" (UniqueName: \"kubernetes.io/projected/c9bd0504-dd4b-41e5-859a-bf8b54b570af-kube-api-access-h9v6g\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7l7t\" (UniqueName: \"kubernetes.io/projected/1b044814-4a86-46db-a651-45578687a328-kube-api-access-j7l7t\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-socket-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.579434    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-sys-fs\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.579568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56249eea-b34a-4073-968c-a9e93e64e999-tmp-dir\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.579507    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret podName:6956353c-c03e-4a3c-8a33-a098f4c11e35 nodeName:}" failed. No retries permitted until 2026-07-24 12:52:58.079490973 +0000 UTC m=+2.068809650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret") pod "global-pull-secret-syncer-shckb" (UID: "6956353c-c03e-4a3c-8a33-a098f4c11e35") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-env-overrides\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-ovn\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-ovnkube-script-lib\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3888e8a-388c-496d-bdbc-8f266fb32067-konnectivity-ca\") pod \"konnectivity-agent-stggw\" (UID: \"b3888e8a-388c-496d-bdbc-8f266fb32067\") " pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-ovn\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-socket-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-sys-fs\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-node-log\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-registration-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9bd0504-dd4b-41e5-859a-bf8b54b570af-iptables-alerter-script\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56249eea-b34a-4073-968c-a9e93e64e999-tmp-dir\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-node-log\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-registration-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.580410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-cni-netd\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6956353c-c03e-4a3c-8a33-a098f4c11e35-dbus\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.579958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5pw\" (UniqueName: \"kubernetes.io/projected/2e1bc47e-ead0-460e-92ba-e57dfafcde04-kube-api-access-sh5pw\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-cni-netd\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-run-ovn-kubernetes\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6956353c-c03e-4a3c-8a33-a098f4c11e35-dbus\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-device-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bca3aa58-22c0-419a-8398-3b1a67b55224-device-dir\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-openvswitch\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-systemd\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-run-ovn-kubernetes\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-run-systemd\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-log-socket\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-log-socket\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.580959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9bd0504-dd4b-41e5-859a-bf8b54b570af-host-slash\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-slash\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-ovnkube-config\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9bd0504-dd4b-41e5-859a-bf8b54b570af-host-slash\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-run-netns\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-slash\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6956353c-c03e-4a3c-8a33-a098f4c11e35-kubelet-config\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9bd0504-dd4b-41e5-859a-bf8b54b570af-iptables-alerter-script\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56249eea-b34a-4073-968c-a9e93e64e999-hosts-file\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b044814-4a86-46db-a651-45578687a328-host-run-netns\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6956353c-c03e-4a3c-8a33-a098f4c11e35-kubelet-config\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56249eea-b34a-4073-968c-a9e93e64e999-hosts-file\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.581544 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.580840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b044814-4a86-46db-a651-45578687a328-ovnkube-config\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.582037 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.582022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b044814-4a86-46db-a651-45578687a328-ovn-node-metrics-cert\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.582194 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.582179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3888e8a-388c-496d-bdbc-8f266fb32067-agent-certs\") pod \"konnectivity-agent-stggw\" (UID: \"b3888e8a-388c-496d-bdbc-8f266fb32067\") " pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:52:57.587372 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.587351    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:52:57.587489 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.587375    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:52:57.587489 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.587397    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nlrsn for pod openshift-network-diagnostics/network-check-target-nn8v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:52:57.587489 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:57.587455    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn podName:d90b3a00-d6a3-457f-a8bb-5f2bde5754b1 nodeName:}" failed. No retries permitted until 2026-07-24 12:52:58.08743756 +0000 UTC m=+2.076756249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nlrsn" (UniqueName: "kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn") pod "network-check-target-nn8v9" (UID: "d90b3a00-d6a3-457f-a8bb-5f2bde5754b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:52:57.588770 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.588747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4xx\" (UniqueName: \"kubernetes.io/projected/bca3aa58-22c0-419a-8398-3b1a67b55224-kube-api-access-xc4xx\") pod \"aws-ebs-csi-driver-node-fqkk4\" (UID: \"bca3aa58-22c0-419a-8398-3b1a67b55224\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.588770 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.588759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7l7t\" (UniqueName: \"kubernetes.io/projected/1b044814-4a86-46db-a651-45578687a328-kube-api-access-j7l7t\") pod \"ovnkube-node-x2pkb\" (UID: \"1b044814-4a86-46db-a651-45578687a328\") " pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.589260 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.589239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vcj6\" (UniqueName: \"kubernetes.io/projected/56249eea-b34a-4073-968c-a9e93e64e999-kube-api-access-9vcj6\") pod \"node-resolver-qf5nm\" (UID: \"56249eea-b34a-4073-968c-a9e93e64e999\") " pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.590232 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.590214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9v6g\" (UniqueName: \"kubernetes.io/projected/c9bd0504-dd4b-41e5-859a-bf8b54b570af-kube-api-access-h9v6g\") pod \"iptables-alerter-v5kqw\" (UID: \"c9bd0504-dd4b-41e5-859a-bf8b54b570af\") " pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.590690 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.590674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5pw\" (UniqueName: \"kubernetes.io/projected/2e1bc47e-ead0-460e-92ba-e57dfafcde04-kube-api-access-sh5pw\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:57.691896 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.691794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tktw2"
Jul 24 12:52:57.701071 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.701045    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82751748_2f8d_49e3_b9ae_3769ec465016.slice/crio-a78bf01cfdf3ff5ff6d3cbfb862394915342a31cb6a01a947fe64fed74a155a5 WatchSource:0}: Error finding container a78bf01cfdf3ff5ff6d3cbfb862394915342a31cb6a01a947fe64fed74a155a5: Status 404 returned error can't find the container with id a78bf01cfdf3ff5ff6d3cbfb862394915342a31cb6a01a947fe64fed74a155a5
Jul 24 12:52:57.703067 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.703050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zl565"
Jul 24 12:52:57.710712 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.710689    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39d5644_41ba_439f_87e2_72f2dfae3422.slice/crio-2cbe36d863ecd5b98a0152c6f2927233aa789b58aba87cb2f5592a09d758e20f WatchSource:0}: Error finding container 2cbe36d863ecd5b98a0152c6f2927233aa789b58aba87cb2f5592a09d758e20f: Status 404 returned error can't find the container with id 2cbe36d863ecd5b98a0152c6f2927233aa789b58aba87cb2f5592a09d758e20f
Jul 24 12:52:57.722358 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.722334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-72tx8"
Jul 24 12:52:57.730163 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.730130    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e2ce4c_941e_4532_9ef8_27420e18a33b.slice/crio-dc5edce8ce43cdb3c0d614d878f8c7aa67fddd59befc67a1374f1e8856efb8ef WatchSource:0}: Error finding container dc5edce8ce43cdb3c0d614d878f8c7aa67fddd59befc67a1374f1e8856efb8ef: Status 404 returned error can't find the container with id dc5edce8ce43cdb3c0d614d878f8c7aa67fddd59befc67a1374f1e8856efb8ef
Jul 24 12:52:57.736446 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.736422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ldqh9"
Jul 24 12:52:57.742665 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.742636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v5kqw"
Jul 24 12:52:57.743912 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.743888    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb5fd7a_722e_4d5a_98da_135ee39ff1a3.slice/crio-f6b06c59a17de58e9b65a6978fd486e28e4a8f761ae42debafc17813541a38a1 WatchSource:0}: Error finding container f6b06c59a17de58e9b65a6978fd486e28e4a8f761ae42debafc17813541a38a1: Status 404 returned error can't find the container with id f6b06c59a17de58e9b65a6978fd486e28e4a8f761ae42debafc17813541a38a1
Jul 24 12:52:57.751486 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.751459    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bd0504_dd4b_41e5_859a_bf8b54b570af.slice/crio-1b62f21a5dac7f7676674c3264b9300ed18bd0c9ee34151734ee2fce2852b1de WatchSource:0}: Error finding container 1b62f21a5dac7f7676674c3264b9300ed18bd0c9ee34151734ee2fce2852b1de: Status 404 returned error can't find the container with id 1b62f21a5dac7f7676674c3264b9300ed18bd0c9ee34151734ee2fce2852b1de
Jul 24 12:52:57.774508 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.774478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:52:57.783543 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.783522    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b044814_4a86_46db_a651_45578687a328.slice/crio-5b01b41ae94abd1b991e36ce98c196aaae1b5cb284c609ec9fc286e479e7b9de WatchSource:0}: Error finding container 5b01b41ae94abd1b991e36ce98c196aaae1b5cb284c609ec9fc286e479e7b9de: Status 404 returned error can't find the container with id 5b01b41ae94abd1b991e36ce98c196aaae1b5cb284c609ec9fc286e479e7b9de
Jul 24 12:52:57.787540 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.787522    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:52:57.792994 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.792972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:52:57.800692 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.800671    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3888e8a_388c_496d_bdbc_8f266fb32067.slice/crio-4f1688d234be39e0a6ac70315484985b500eb1801fee7275323a5651be11eaed WatchSource:0}: Error finding container 4f1688d234be39e0a6ac70315484985b500eb1801fee7275323a5651be11eaed: Status 404 returned error can't find the container with id 4f1688d234be39e0a6ac70315484985b500eb1801fee7275323a5651be11eaed
Jul 24 12:52:57.821268 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.821244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4"
Jul 24 12:52:57.825384 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:57.825362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qf5nm"
Jul 24 12:52:57.829072 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.829043    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca3aa58_22c0_419a_8398_3b1a67b55224.slice/crio-caf36c8a4c66ef2b9cfcc1eba9bd1a510ee6798a2fac728b08a7601983f05ca2 WatchSource:0}: Error finding container caf36c8a4c66ef2b9cfcc1eba9bd1a510ee6798a2fac728b08a7601983f05ca2: Status 404 returned error can't find the container with id caf36c8a4c66ef2b9cfcc1eba9bd1a510ee6798a2fac728b08a7601983f05ca2
Jul 24 12:52:57.833089 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:52:57.833066    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56249eea_b34a_4073_968c_a9e93e64e999.slice/crio-97d915fd9e31b901e5bc439b929512a874265ac5b4f2c575d1a816dd6d0ade52 WatchSource:0}: Error finding container 97d915fd9e31b901e5bc439b929512a874265ac5b4f2c575d1a816dd6d0ade52: Status 404 returned error can't find the container with id 97d915fd9e31b901e5bc439b929512a874265ac5b4f2c575d1a816dd6d0ade52
Jul 24 12:52:58.083710 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.083615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:58.083860 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.083715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:58.083860 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:58.083799    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:52:58.083860 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:58.083811    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:52:58.084035 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:58.083879    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret podName:6956353c-c03e-4a3c-8a33-a098f4c11e35 nodeName:}" failed. No retries permitted until 2026-07-24 12:52:59.083858131 +0000 UTC m=+3.073176808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret") pod "global-pull-secret-syncer-shckb" (UID: "6956353c-c03e-4a3c-8a33-a098f4c11e35") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:52:58.084035 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:58.083900    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs podName:2e1bc47e-ead0-460e-92ba-e57dfafcde04 nodeName:}" failed. No retries permitted until 2026-07-24 12:52:59.083889425 +0000 UTC m=+3.073208102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs") pod "network-metrics-daemon-8nqcd" (UID: "2e1bc47e-ead0-460e-92ba-e57dfafcde04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:52:58.185210 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.185173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:52:58.185400 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:58.185363    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:52:58.185497 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:58.185399    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:52:58.185497 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:58.185415    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nlrsn for pod openshift-network-diagnostics/network-check-target-nn8v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:52:58.185497 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:58.185483    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn podName:d90b3a00-d6a3-457f-a8bb-5f2bde5754b1 nodeName:}" failed. No retries permitted until 2026-07-24 12:52:59.185462726 +0000 UTC m=+3.174781406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlrsn" (UniqueName: "kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn") pod "network-check-target-nn8v9" (UID: "d90b3a00-d6a3-457f-a8bb-5f2bde5754b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:52:58.414431 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.414307    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 12:47:57 +0000 UTC" deadline="2028-01-04 11:28:54.141170197 +0000 UTC"
Jul 24 12:52:58.414431 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.414391    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12694h35m55.72678481s"
Jul 24 12:52:58.540146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.539878    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:52:58.579919 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.579878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-stggw" event={"ID":"b3888e8a-388c-496d-bdbc-8f266fb32067","Type":"ContainerStarted","Data":"4f1688d234be39e0a6ac70315484985b500eb1801fee7275323a5651be11eaed"}
Jul 24 12:52:58.585470 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.585329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"5b01b41ae94abd1b991e36ce98c196aaae1b5cb284c609ec9fc286e479e7b9de"}
Jul 24 12:52:58.590390 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.590197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zl565" event={"ID":"a39d5644-41ba-439f-87e2-72f2dfae3422","Type":"ContainerStarted","Data":"2cbe36d863ecd5b98a0152c6f2927233aa789b58aba87cb2f5592a09d758e20f"}
Jul 24 12:52:58.592515 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.592441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v5kqw" event={"ID":"c9bd0504-dd4b-41e5-859a-bf8b54b570af","Type":"ContainerStarted","Data":"1b62f21a5dac7f7676674c3264b9300ed18bd0c9ee34151734ee2fce2852b1de"}
Jul 24 12:52:58.602042 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.600413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" event={"ID":"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3","Type":"ContainerStarted","Data":"f6b06c59a17de58e9b65a6978fd486e28e4a8f761ae42debafc17813541a38a1"}
Jul 24 12:52:58.616170 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.616125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-72tx8" event={"ID":"a0e2ce4c-941e-4532-9ef8-27420e18a33b","Type":"ContainerStarted","Data":"dc5edce8ce43cdb3c0d614d878f8c7aa67fddd59befc67a1374f1e8856efb8ef"}
Jul 24 12:52:58.619876 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.619816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tktw2" event={"ID":"82751748-2f8d-49e3-b9ae-3769ec465016","Type":"ContainerStarted","Data":"a78bf01cfdf3ff5ff6d3cbfb862394915342a31cb6a01a947fe64fed74a155a5"}
Jul 24 12:52:58.631048 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.630615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qf5nm" event={"ID":"56249eea-b34a-4073-968c-a9e93e64e999","Type":"ContainerStarted","Data":"97d915fd9e31b901e5bc439b929512a874265ac5b4f2c575d1a816dd6d0ade52"}
Jul 24 12:52:58.634595 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:58.634509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4" event={"ID":"bca3aa58-22c0-419a-8398-3b1a67b55224","Type":"ContainerStarted","Data":"caf36c8a4c66ef2b9cfcc1eba9bd1a510ee6798a2fac728b08a7601983f05ca2"}
Jul 24 12:52:59.092385 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.092344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:59.092562 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.092413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:59.092637 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.092581    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:52:59.092689 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.092652    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret podName:6956353c-c03e-4a3c-8a33-a098f4c11e35 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:01.092632892 +0000 UTC m=+5.081951585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret") pod "global-pull-secret-syncer-shckb" (UID: "6956353c-c03e-4a3c-8a33-a098f4c11e35") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:52:59.093137 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.093103    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:52:59.093268 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.093171    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs podName:2e1bc47e-ead0-460e-92ba-e57dfafcde04 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:01.093156995 +0000 UTC m=+5.082475686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs") pod "network-metrics-daemon-8nqcd" (UID: "2e1bc47e-ead0-460e-92ba-e57dfafcde04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:52:59.193901 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.193816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:52:59.194098 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.194030    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:52:59.194098 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.194051    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:52:59.194098 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.194065    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nlrsn for pod openshift-network-diagnostics/network-check-target-nn8v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:52:59.194277 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.194130    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn podName:d90b3a00-d6a3-457f-a8bb-5f2bde5754b1 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:01.194110369 +0000 UTC m=+5.183429058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlrsn" (UniqueName: "kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn") pod "network-check-target-nn8v9" (UID: "d90b3a00-d6a3-457f-a8bb-5f2bde5754b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:52:59.415361 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.415307    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 12:47:57 +0000 UTC" deadline="2027-12-30 01:32:48.75609183 +0000 UTC"
Jul 24 12:52:59.415361 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.415352    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12564h39m49.340743565s"
Jul 24 12:52:59.556992 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.556886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:52:59.557179 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.557052    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:52:59.557179 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.557141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:52:59.557297 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.557215    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:52:59.557297 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.557284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:52:59.557397 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:52:59.557371    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:52:59.646372 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:52:59.646322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal" event={"ID":"630256be83544d972c8b0ff554f26b54","Type":"ContainerStarted","Data":"5ed6b32f82835472a571af0a39e5a990be32b513b0797dd5e526539b87906801"}
Jul 24 12:53:01.117152 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:01.116277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:01.117152 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:01.116340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:01.117152 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.116513    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:53:01.117152 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.116584    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret podName:6956353c-c03e-4a3c-8a33-a098f4c11e35 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:05.11656478 +0000 UTC m=+9.105883473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret") pod "global-pull-secret-syncer-shckb" (UID: "6956353c-c03e-4a3c-8a33-a098f4c11e35") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:53:01.117152 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.117034    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:53:01.117152 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.117089    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs podName:2e1bc47e-ead0-460e-92ba-e57dfafcde04 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:05.117072268 +0000 UTC m=+9.106390943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs") pod "network-metrics-daemon-8nqcd" (UID: "2e1bc47e-ead0-460e-92ba-e57dfafcde04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:53:01.217535 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:01.217490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:01.217778 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.217738    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:53:01.217778 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.217765    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:53:01.217778 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.217779    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nlrsn for pod openshift-network-diagnostics/network-check-target-nn8v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:53:01.217953 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.217850    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn podName:d90b3a00-d6a3-457f-a8bb-5f2bde5754b1 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:05.217828215 +0000 UTC m=+9.207146895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlrsn" (UniqueName: "kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn") pod "network-check-target-nn8v9" (UID: "d90b3a00-d6a3-457f-a8bb-5f2bde5754b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:53:01.557223 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:01.557128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:01.557423 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.557278    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:01.557721 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:01.557699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:01.557843 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.557820    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:01.557959 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:01.557881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:01.558045 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:01.557958    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:03.557782 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:03.557740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:03.557782 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:03.557782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:03.558319 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:03.557744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:03.558319 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:03.557872    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:03.558319 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:03.557961    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:03.558319 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:03.558082    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:05.156095 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:05.155439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:05.156095 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:05.155502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:05.156095 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.155626    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:53:05.156095 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.155679    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret podName:6956353c-c03e-4a3c-8a33-a098f4c11e35 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:13.155665419 +0000 UTC m=+17.144984093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret") pod "global-pull-secret-syncer-shckb" (UID: "6956353c-c03e-4a3c-8a33-a098f4c11e35") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:53:05.156095 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.156035    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:53:05.156095 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.156070    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs podName:2e1bc47e-ead0-460e-92ba-e57dfafcde04 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:13.156059668 +0000 UTC m=+17.145378341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs") pod "network-metrics-daemon-8nqcd" (UID: "2e1bc47e-ead0-460e-92ba-e57dfafcde04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:53:05.256870 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:05.256226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:05.256870 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.256482    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:53:05.256870 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.256498    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:53:05.256870 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.256508    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nlrsn for pod openshift-network-diagnostics/network-check-target-nn8v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:53:05.256870 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.256560    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn podName:d90b3a00-d6a3-457f-a8bb-5f2bde5754b1 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:13.256546111 +0000 UTC m=+17.245864785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlrsn" (UniqueName: "kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn") pod "network-check-target-nn8v9" (UID: "d90b3a00-d6a3-457f-a8bb-5f2bde5754b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:53:05.557214 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:05.557147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:05.557356 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.557262    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:05.557675 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:05.557654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:05.557849 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.557759    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:05.557849 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:05.557809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:05.557992 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:05.557875    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:07.558237 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:07.557743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:07.558237 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:07.557743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:07.558237 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:07.557899    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:07.558237 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:07.558006    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:07.558237 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:07.557757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:07.558237 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:07.558114    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:09.557656 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:09.557620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:09.558120 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:09.557620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:09.558120 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:09.557732    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:09.558120 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:09.557818    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:09.558120 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:09.557635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:09.558120 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:09.557911    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:11.557490 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:11.557446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:11.557953 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:11.557446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:11.557953 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:11.557585    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:11.557953 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:11.557682    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:11.557953 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:11.557446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:11.557953 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:11.557785    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:13.222241 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:13.222144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:13.222241 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:13.222239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:13.222696 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.222350    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:53:13.222696 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.222360    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:53:13.222696 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.222424    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret podName:6956353c-c03e-4a3c-8a33-a098f4c11e35 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:29.222403114 +0000 UTC m=+33.211721792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret") pod "global-pull-secret-syncer-shckb" (UID: "6956353c-c03e-4a3c-8a33-a098f4c11e35") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:53:13.222696 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.222443    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs podName:2e1bc47e-ead0-460e-92ba-e57dfafcde04 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:29.222433215 +0000 UTC m=+33.211751896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs") pod "network-metrics-daemon-8nqcd" (UID: "2e1bc47e-ead0-460e-92ba-e57dfafcde04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:53:13.323260 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:13.323225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:13.323440 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.323374    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:53:13.323440 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.323396    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:53:13.323440 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.323409    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nlrsn for pod openshift-network-diagnostics/network-check-target-nn8v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:53:13.323569 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.323476    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn podName:d90b3a00-d6a3-457f-a8bb-5f2bde5754b1 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:29.323454095 +0000 UTC m=+33.312772785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlrsn" (UniqueName: "kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn") pod "network-check-target-nn8v9" (UID: "d90b3a00-d6a3-457f-a8bb-5f2bde5754b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:53:13.557002 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:13.556904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:13.557152 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:13.556903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:13.557152 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.557056    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:13.557152 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:13.556911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:13.557152 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.557147    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:13.557344 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:13.557192    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:15.556977 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:15.556939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:15.556977 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:15.556966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:15.557415 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:15.557069    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:15.557415 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:15.557110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:15.557415 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:15.557228    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:15.557415 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:15.557317    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:17.557285 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:17.557254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:17.557665 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:17.557258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:17.557665 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:17.557362    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:17.557665 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:17.557443    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:17.557665 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:17.557256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:17.557665 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:17.557560    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:18.684746 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.684483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qf5nm" event={"ID":"56249eea-b34a-4073-968c-a9e93e64e999","Type":"ContainerStarted","Data":"826e1158ba812727bf5dd52190ed675b58e239029a486b18851d455cc6a38a42"}
Jul 24 12:53:18.685869 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.685846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4" event={"ID":"bca3aa58-22c0-419a-8398-3b1a67b55224","Type":"ContainerStarted","Data":"f65284cdcc498eb3eb3003e0fe4181a8a93afc2cabf5629ff1693208e6c9db19"}
Jul 24 12:53:18.687325 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.687299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-stggw" event={"ID":"b3888e8a-388c-496d-bdbc-8f266fb32067","Type":"ContainerStarted","Data":"3770abaeac52cdee9d721b0455f71639ab2b15aea43db02d2559cda93b82cbff"}
Jul 24 12:53:18.690984 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.690961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"83d336c8b43ce9eade8b8a9d5025b762e26140ca283973eea7ab9b66f68e8ef9"}
Jul 24 12:53:18.691090 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.691029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"94ddc83c768f4252877e7c44696f7980990aac34a66e090a0a109b720dfd0a8f"}
Jul 24 12:53:18.691090 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.691059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"36f0d68eecb4a4614c3342110d03cb78109dcc5fff19e59ed753d277f9475abe"}
Jul 24 12:53:18.691090 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.691087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"d12821db839ccdd83f5083d9b8ad7781cce7617f23d3e2345a82abac38ff2f87"}
Jul 24 12:53:18.691244 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.691121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"bd5882aa7981a78f05fb867f5a18ca07ca7fb4e6d51d2293088f2a6853e0d19f"}
Jul 24 12:53:18.691244 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.691149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"e65c9e2ff4670bf340c6afc725c48bed1480f7c9e3cccbc343f215d6ab589c1e"}
Jul 24 12:53:18.692514 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.692487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zl565" event={"ID":"a39d5644-41ba-439f-87e2-72f2dfae3422","Type":"ContainerStarted","Data":"cf409dcfbdfebf91947e16c0cda932fa71844489c070ecdd1535f24ce4bd40fa"}
Jul 24 12:53:18.694029 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.694006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal" event={"ID":"630256be83544d972c8b0ff554f26b54","Type":"ContainerStarted","Data":"24009aaf0da549e2ed52c09166624123c9e6a1ff15db0019c1a2fda9166ed7ee"}
Jul 24 12:53:18.695548 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.695525    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bb5fd7a-722e-4d5a-98da-135ee39ff1a3" containerID="bc4207f09902f1fe523c3c9997e8459f79aeaedd8eddf844d46e4c0ddb86c6a0" exitCode=0
Jul 24 12:53:18.695654 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.695590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" event={"ID":"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3","Type":"ContainerDied","Data":"bc4207f09902f1fe523c3c9997e8459f79aeaedd8eddf844d46e4c0ddb86c6a0"}
Jul 24 12:53:18.697483 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.697460    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-72tx8" event={"ID":"a0e2ce4c-941e-4532-9ef8-27420e18a33b","Type":"ContainerStarted","Data":"dfaff62253835c5fc0c28ffd21a34007c55a3de3be7ec243b38b864685c7dbe2"}
Jul 24 12:53:18.698941 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.698896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tktw2" event={"ID":"82751748-2f8d-49e3-b9ae-3769ec465016","Type":"ContainerStarted","Data":"afa43f9d84ba3933128cdf678c2ae51e9076870f14c151cecc1238b4af9c2aac"}
Jul 24 12:53:18.700638 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.700618    2579 generic.go:356] "Generic (PLEG): container finished" podID="61a709d601992dbacaba1dc40727bd91" containerID="1b8334ce777ef0c5204e99989540fa15e4ee0859690a1b3da50662ccf41aa0e3" exitCode=0
Jul 24 12:53:18.700741 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.700649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal" event={"ID":"61a709d601992dbacaba1dc40727bd91","Type":"ContainerDied","Data":"1b8334ce777ef0c5204e99989540fa15e4ee0859690a1b3da50662ccf41aa0e3"}
Jul 24 12:53:18.703311 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.703276    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qf5nm" podStartSLOduration=1.771478298 podStartE2EDuration="21.703264576s" totalImagesPullingTime="19.931786278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:18.702963329 +0000 UTC m=+22.692282016" watchObservedRunningTime="2026-07-24 12:53:18.703264576 +0000 UTC m=+22.692583272"
Jul 24 12:53:18.724397 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.724362    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-stggw" podStartSLOduration=6.563431101 podStartE2EDuration="21.724347823s" totalImagesPullingTime="15.160916722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:18.724339776 +0000 UTC m=+22.713658472" watchObservedRunningTime="2026-07-24 12:53:18.724347823 +0000 UTC m=+22.713666519"
Jul 24 12:53:18.747215 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.747155    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-72tx8" podStartSLOduration=2.71260485 podStartE2EDuration="22.747135212s" totalImagesPullingTime="20.034530362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:18.74646759 +0000 UTC m=+22.735786287" watchObservedRunningTime="2026-07-24 12:53:18.747135212 +0000 UTC m=+22.736453908"
Jul 24 12:53:18.878198 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.878147    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-29.ec2.internal" podStartSLOduration=21.878129812 podStartE2EDuration="21.878129812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:18.849852282 +0000 UTC m=+22.839170978" watchObservedRunningTime="2026-07-24 12:53:18.878129812 +0000 UTC m=+22.867448516"
Jul 24 12:53:18.878400 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.878325    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tktw2" podStartSLOduration=2.8133987339999997 podStartE2EDuration="22.878315385s" totalImagesPullingTime="20.064916651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:18.877745184 +0000 UTC m=+22.867063872" watchObservedRunningTime="2026-07-24 12:53:18.878315385 +0000 UTC m=+22.867634082"
Jul 24 12:53:18.899939 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:18.899878    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zl565" podStartSLOduration=2.8475774190000003 podStartE2EDuration="22.899861327s" totalImagesPullingTime="20.052283908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:18.89945131 +0000 UTC m=+22.888770010" watchObservedRunningTime="2026-07-24 12:53:18.899861327 +0000 UTC m=+22.889180023"
Jul 24 12:53:19.334613 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.334586    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 12:53:19.440142 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.439895    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T12:53:19.334606288Z","UUID":"693e282a-039d-47f2-bdd9-444a0268b6f2","Handler":null,"Name":"","Endpoint":""}
Jul 24 12:53:19.441686 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.441660    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 12:53:19.441813 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.441707    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 12:53:19.557065 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.557034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:19.557334 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.557034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:19.557334 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:19.557172    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:19.557334 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.557035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:19.557334 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:19.557224    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:19.557334 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:19.557324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:19.704825 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.704748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal" event={"ID":"61a709d601992dbacaba1dc40727bd91","Type":"ContainerStarted","Data":"d00b04941871fe712346ac34f464e8be7cb8e8c82f58bc9bf8225b2ccc175adc"}
Jul 24 12:53:19.706990 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.706962    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4" event={"ID":"bca3aa58-22c0-419a-8398-3b1a67b55224","Type":"ContainerStarted","Data":"582e40c26670b4bc883269b01febb1d8e411993b4733f38830f2c8eaf763d487"}
Jul 24 12:53:19.708527 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.708497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v5kqw" event={"ID":"c9bd0504-dd4b-41e5-859a-bf8b54b570af","Type":"ContainerStarted","Data":"decefda26a15d4767026511346938d28a82db54b7fb09a5b673501d0fd8770ea"}
Jul 24 12:53:19.728591 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:19.728546    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-29.ec2.internal" podStartSLOduration=22.728530754 podStartE2EDuration="22.728530754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:19.728001483 +0000 UTC m=+23.717320179" watchObservedRunningTime="2026-07-24 12:53:19.728530754 +0000 UTC m=+23.717849449"
Jul 24 12:53:20.716884 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:20.716855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"0e7cfb3878634e386437a9f81e6852c0b8e8d697a03815f1bb430018e3563249"}
Jul 24 12:53:21.557259 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:21.557225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:21.557463 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:21.557268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:21.557463 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:21.557232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:21.557463 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:21.557343    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:21.557463 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:21.557399    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:21.557463 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:21.557456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:21.721350 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:21.721269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4" event={"ID":"bca3aa58-22c0-419a-8398-3b1a67b55224","Type":"ContainerStarted","Data":"fa3874467ea12550b47292cdd9deef909c2931033ce771548d66234e5ca1f049"}
Jul 24 12:53:21.742829 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:21.742772    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqkk4" podStartSLOduration=1.780421474 podStartE2EDuration="24.742749787s" totalImagesPullingTime="22.962328313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:21.742697107 +0000 UTC m=+25.732015803" watchObservedRunningTime="2026-07-24 12:53:21.742749787 +0000 UTC m=+25.732068482"
Jul 24 12:53:21.743431 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:21.743395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v5kqw" podStartSLOduration=4.731902175 podStartE2EDuration="24.743381805s" totalImagesPullingTime="20.01147963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:19.753112491 +0000 UTC m=+23.742431187" watchObservedRunningTime="2026-07-24 12:53:21.743381805 +0000 UTC m=+25.732700501"
Jul 24 12:53:21.751844 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:21.751810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:53:21.752429 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:21.752402    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:53:22.723905 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:22.723846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:53:22.724313 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:22.724265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-stggw"
Jul 24 12:53:23.557500 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.557275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:23.557648 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.557275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:23.557648 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:23.557555    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:23.557648 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.557271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:23.557648 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:23.557637    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:23.557795 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:23.557689    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:23.729332 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.729298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" event={"ID":"1b044814-4a86-46db-a651-45578687a328","Type":"ContainerStarted","Data":"04039848644fc4a952d83ac6b550a387c4ce51607880541543b985ff0022fbea"}
Jul 24 12:53:23.729895 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.729532    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:53:23.731212 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.731186    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bb5fd7a-722e-4d5a-98da-135ee39ff1a3" containerID="97dfe223754a8f0a1d1a197debe6a2e142f7dca31890c76d552c455020faa70f" exitCode=0
Jul 24 12:53:23.731328 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.731261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" event={"ID":"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3","Type":"ContainerDied","Data":"97dfe223754a8f0a1d1a197debe6a2e142f7dca31890c76d552c455020faa70f"}
Jul 24 12:53:23.745917 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.745896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:53:23.760484 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:23.760441    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb" podStartSLOduration=6.7777639480000005 podStartE2EDuration="26.760428055s" totalImagesPullingTime="19.982664107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:23.758700696 +0000 UTC m=+27.748019391" watchObservedRunningTime="2026-07-24 12:53:23.760428055 +0000 UTC m=+27.749746751"
Jul 24 12:53:24.733361 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:24.733280    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 12:53:24.733919 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:24.733688    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:53:24.748811 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:24.748784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:53:24.794386 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:24.794360    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:53:25.557243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.557063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:25.557442 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.557063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:25.557442 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:25.557319    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:25.557442 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.557121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:25.557578 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:25.557395    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:25.557578 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:25.557501    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:25.658465 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.658412    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8nqcd"]
Jul 24 12:53:25.659156 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.659127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-shckb"]
Jul 24 12:53:25.660503 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.660481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nn8v9"]
Jul 24 12:53:25.737976 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.737882    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bb5fd7a-722e-4d5a-98da-135ee39ff1a3" containerID="5f23b5b8f84e404364c9a475ad424235fff31c163732753cbf13333c2cbf36d3" exitCode=0
Jul 24 12:53:25.737976 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.737942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" event={"ID":"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3","Type":"ContainerDied","Data":"5f23b5b8f84e404364c9a475ad424235fff31c163732753cbf13333c2cbf36d3"}
Jul 24 12:53:25.738358 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.738034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:25.738358 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.738038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:25.738358 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:25.738050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:25.738358 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:25.738116    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:25.738358 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:25.738238    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:25.738743 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:25.738707    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:27.557205 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:27.557109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:27.557609 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:27.557225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:27.557609 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:27.557120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:27.557609 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:27.557128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:27.557609 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:27.557349    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:27.557609 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:27.557427    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:27.745255 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:27.745222    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bb5fd7a-722e-4d5a-98da-135ee39ff1a3" containerID="749493a53f033b5f37f6bde0b62b50148d3828bfc70a063aae9fb9fd8c2a4b81" exitCode=0
Jul 24 12:53:27.745447 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:27.745270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" event={"ID":"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3","Type":"ContainerDied","Data":"749493a53f033b5f37f6bde0b62b50148d3828bfc70a063aae9fb9fd8c2a4b81"}
Jul 24 12:53:29.238242 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:29.237975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:29.238242 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:29.238235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:29.238687 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.238159    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:53:29.238687 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.238432    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs podName:2e1bc47e-ead0-460e-92ba-e57dfafcde04 nodeName:}" failed. No retries permitted until 2026-07-24 12:54:01.238419333 +0000 UTC m=+65.227738007 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs") pod "network-metrics-daemon-8nqcd" (UID: "2e1bc47e-ead0-460e-92ba-e57dfafcde04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:53:29.238779 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.238376    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:53:29.238779 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.238767    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret podName:6956353c-c03e-4a3c-8a33-a098f4c11e35 nodeName:}" failed. No retries permitted until 2026-07-24 12:54:01.238749775 +0000 UTC m=+65.228068452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret") pod "global-pull-secret-syncer-shckb" (UID: "6956353c-c03e-4a3c-8a33-a098f4c11e35") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:53:29.339568 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:29.339530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:29.339766 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.339749    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:53:29.339835 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.339772    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:53:29.339835 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.339782    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nlrsn for pod openshift-network-diagnostics/network-check-target-nn8v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:53:29.339951 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.339880    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn podName:d90b3a00-d6a3-457f-a8bb-5f2bde5754b1 nodeName:}" failed. No retries permitted until 2026-07-24 12:54:01.339863678 +0000 UTC m=+65.329182353 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlrsn" (UniqueName: "kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn") pod "network-check-target-nn8v9" (UID: "d90b3a00-d6a3-457f-a8bb-5f2bde5754b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:53:29.557347 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:29.557241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:29.557347 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:29.557268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:29.557347 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:29.557292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:29.557620 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.557399    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shckb" podUID="6956353c-c03e-4a3c-8a33-a098f4c11e35"
Jul 24 12:53:29.557620 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.557526    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nqcd" podUID="2e1bc47e-ead0-460e-92ba-e57dfafcde04"
Jul 24 12:53:29.557620 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:29.557594    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nn8v9" podUID="d90b3a00-d6a3-457f-a8bb-5f2bde5754b1"
Jul 24 12:53:30.872207 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.872167    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-29.ec2.internal" event="NodeReady"
Jul 24 12:53:30.872606 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.872337    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 12:53:30.945366 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.945333    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-745ff49dc7-zm67b"]
Jul 24 12:53:30.947562 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.947538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:30.950942 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.950907    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 12:53:30.951699 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.951681    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 12:53:30.952182 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.952169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 12:53:30.953938 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.953901    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zfr64\""
Jul 24 12:53:30.960253 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.960222    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 12:53:30.964370 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.964348    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hp2hl"]
Jul 24 12:53:30.966428 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.966408    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8mj9k"]
Jul 24 12:53:30.966530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.966518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:30.968340 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.968315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8mj9k"
Jul 24 12:53:30.969519 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.969497    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7xh6d\""
Jul 24 12:53:30.969739 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.969724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 12:53:30.969819 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.969750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-745ff49dc7-zm67b"]
Jul 24 12:53:30.970036 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.970017    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 12:53:30.971735 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.971718    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 12:53:30.972569 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.972553    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 12:53:30.972810 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.972794    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxgfg\""
Jul 24 12:53:30.973089 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.973073    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 12:53:30.984767 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.984741    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8mj9k"]
Jul 24 12:53:30.991967 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:30.991939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hp2hl"]
Jul 24 12:53:31.051714 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.051679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-trusted-ca\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.051882 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.051734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-bound-sa-token\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.051882 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.051761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbnl\" (UniqueName: \"kubernetes.io/projected/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-kube-api-access-7bbnl\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.051882 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.051839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.051882 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.051869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.052122 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.051952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-installation-pull-secrets\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.052122 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.051981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f8p4\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-kube-api-access-2f8p4\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.052122 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.051996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-config-volume\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.052122 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.052016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-tmp-dir\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.052122 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.052064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-image-registry-private-configuration\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.052122 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.052104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4vl\" (UniqueName: \"kubernetes.io/projected/c42f561d-917e-4912-9c0d-7c7adcbdfb86-kube-api-access-vw4vl\") pod \"ingress-canary-8mj9k\" (UID: \"c42f561d-917e-4912-9c0d-7c7adcbdfb86\") " pod="openshift-ingress-canary/ingress-canary-8mj9k"
Jul 24 12:53:31.052404 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.052144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fbe6497-426b-41d3-a613-a792e28d5d68-ca-trust-extracted\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.052404 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.052157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c42f561d-917e-4912-9c0d-7c7adcbdfb86-cert\") pod \"ingress-canary-8mj9k\" (UID: \"c42f561d-917e-4912-9c0d-7c7adcbdfb86\") " pod="openshift-ingress-canary/ingress-canary-8mj9k"
Jul 24 12:53:31.052404 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.052175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-certificates\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153135 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f8p4\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-kube-api-access-2f8p4\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153135 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-config-volume\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.153135 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-tmp-dir\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-image-registry-private-configuration\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4vl\" (UniqueName: \"kubernetes.io/projected/c42f561d-917e-4912-9c0d-7c7adcbdfb86-kube-api-access-vw4vl\") pod \"ingress-canary-8mj9k\" (UID: \"c42f561d-917e-4912-9c0d-7c7adcbdfb86\") " pod="openshift-ingress-canary/ingress-canary-8mj9k"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fbe6497-426b-41d3-a613-a792e28d5d68-ca-trust-extracted\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c42f561d-917e-4912-9c0d-7c7adcbdfb86-cert\") pod \"ingress-canary-8mj9k\" (UID: \"c42f561d-917e-4912-9c0d-7c7adcbdfb86\") " pod="openshift-ingress-canary/ingress-canary-8mj9k"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-certificates\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-trusted-ca\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-bound-sa-token\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbnl\" (UniqueName: \"kubernetes.io/projected/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-kube-api-access-7bbnl\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.153457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-installation-pull-secrets\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.153946 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.153730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-config-volume\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.154569 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.154535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-certificates\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.154736 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.154709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-trusted-ca\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.154835 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.154798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-tmp-dir\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.154949 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.154913    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:53:31.155051 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.154954    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-745ff49dc7-zm67b: secret "image-registry-tls" not found
Jul 24 12:53:31.155051 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.154964    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:53:31.155051 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.155022    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls podName:3fbe6497-426b-41d3-a613-a792e28d5d68 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:31.655004527 +0000 UTC m=+35.644323200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls") pod "image-registry-745ff49dc7-zm67b" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68") : secret "image-registry-tls" not found
Jul 24 12:53:31.155211 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.155063    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls podName:8354fa4f-cf8e-479d-bfc2-59514dff4b1c nodeName:}" failed. No retries permitted until 2026-07-24 12:53:31.655046955 +0000 UTC m=+35.644365644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls") pod "dns-default-hp2hl" (UID: "8354fa4f-cf8e-479d-bfc2-59514dff4b1c") : secret "dns-default-metrics-tls" not found
Jul 24 12:53:31.155382 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.155357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fbe6497-426b-41d3-a613-a792e28d5d68-ca-trust-extracted\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.161167 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.161144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-installation-pull-secrets\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.161268 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.161229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-image-registry-private-configuration\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.162141 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.162123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c42f561d-917e-4912-9c0d-7c7adcbdfb86-cert\") pod \"ingress-canary-8mj9k\" (UID: \"c42f561d-917e-4912-9c0d-7c7adcbdfb86\") " pod="openshift-ingress-canary/ingress-canary-8mj9k"
Jul 24 12:53:31.164057 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.164039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f8p4\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-kube-api-access-2f8p4\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.164388 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.164369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-bound-sa-token\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.164820 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.164800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbnl\" (UniqueName: \"kubernetes.io/projected/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-kube-api-access-7bbnl\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.172108 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.172085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4vl\" (UniqueName: \"kubernetes.io/projected/c42f561d-917e-4912-9c0d-7c7adcbdfb86-kube-api-access-vw4vl\") pod \"ingress-canary-8mj9k\" (UID: \"c42f561d-917e-4912-9c0d-7c7adcbdfb86\") " pod="openshift-ingress-canary/ingress-canary-8mj9k"
Jul 24 12:53:31.284380 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.284334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8mj9k"
Jul 24 12:53:31.557425 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.557337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:53:31.557594 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.557436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:53:31.557676 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.557658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:53:31.561388 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.561363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lv5pf\""
Jul 24 12:53:31.561551 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.561467    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-txwsv\""
Jul 24 12:53:31.561604 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.561580    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 12:53:31.561694 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.561679    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 12:53:31.561973 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.561958    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 12:53:31.562318 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.562301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 12:53:31.657110 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.657074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:31.657110 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:31.657113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:31.657340 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.657236    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:53:31.657340 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.657243    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:53:31.657340 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.657263    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-745ff49dc7-zm67b: secret "image-registry-tls" not found
Jul 24 12:53:31.657340 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.657303    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls podName:8354fa4f-cf8e-479d-bfc2-59514dff4b1c nodeName:}" failed. No retries permitted until 2026-07-24 12:53:32.657287588 +0000 UTC m=+36.646606262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls") pod "dns-default-hp2hl" (UID: "8354fa4f-cf8e-479d-bfc2-59514dff4b1c") : secret "dns-default-metrics-tls" not found
Jul 24 12:53:31.657340 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:31.657323    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls podName:3fbe6497-426b-41d3-a613-a792e28d5d68 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:32.65731077 +0000 UTC m=+36.646629448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls") pod "image-registry-745ff49dc7-zm67b" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68") : secret "image-registry-tls" not found
Jul 24 12:53:32.220174 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:32.220018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8mj9k"]
Jul 24 12:53:32.223380 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:53:32.223356    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42f561d_917e_4912_9c0d_7c7adcbdfb86.slice/crio-01da7034bfc202034bff028b54f3a3926fcbd87e7cb61b8d75b0dd5ab64b0b52 WatchSource:0}: Error finding container 01da7034bfc202034bff028b54f3a3926fcbd87e7cb61b8d75b0dd5ab64b0b52: Status 404 returned error can't find the container with id 01da7034bfc202034bff028b54f3a3926fcbd87e7cb61b8d75b0dd5ab64b0b52
Jul 24 12:53:32.664994 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:32.664956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:32.665201 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:32.665007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:32.665201 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:32.665093    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:53:32.665201 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:32.665116    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-745ff49dc7-zm67b: secret "image-registry-tls" not found
Jul 24 12:53:32.665201 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:32.665163    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:53:32.665201 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:32.665200    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls podName:3fbe6497-426b-41d3-a613-a792e28d5d68 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:34.665179249 +0000 UTC m=+38.654497925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls") pod "image-registry-745ff49dc7-zm67b" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68") : secret "image-registry-tls" not found
Jul 24 12:53:32.665401 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:32.665218    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls podName:8354fa4f-cf8e-479d-bfc2-59514dff4b1c nodeName:}" failed. No retries permitted until 2026-07-24 12:53:34.665208895 +0000 UTC m=+38.654527572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls") pod "dns-default-hp2hl" (UID: "8354fa4f-cf8e-479d-bfc2-59514dff4b1c") : secret "dns-default-metrics-tls" not found
Jul 24 12:53:32.759834 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:32.759793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8mj9k" event={"ID":"c42f561d-917e-4912-9c0d-7c7adcbdfb86","Type":"ContainerStarted","Data":"01da7034bfc202034bff028b54f3a3926fcbd87e7cb61b8d75b0dd5ab64b0b52"}
Jul 24 12:53:32.762875 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:32.762846    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bb5fd7a-722e-4d5a-98da-135ee39ff1a3" containerID="c5e005eb5e53b187cc0c29507a0a0e71dd3a7258a7cdf7439bae642f41ad339b" exitCode=0
Jul 24 12:53:32.763024 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:32.762900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" event={"ID":"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3","Type":"ContainerDied","Data":"c5e005eb5e53b187cc0c29507a0a0e71dd3a7258a7cdf7439bae642f41ad339b"}
Jul 24 12:53:33.769255 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:33.769220    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bb5fd7a-722e-4d5a-98da-135ee39ff1a3" containerID="ce77497e323c6debd2711a2c147178b6b5038d966142de2dca167f79d76fbc10" exitCode=0
Jul 24 12:53:33.769712 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:33.769305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" event={"ID":"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3","Type":"ContainerDied","Data":"ce77497e323c6debd2711a2c147178b6b5038d966142de2dca167f79d76fbc10"}
Jul 24 12:53:34.681189 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:34.681154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:34.681365 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:34.681199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:34.681365 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:34.681317    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:53:34.681365 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:34.681330    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:53:34.681365 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:34.681351    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-745ff49dc7-zm67b: secret "image-registry-tls" not found
Jul 24 12:53:34.681517 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:34.681373    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls podName:8354fa4f-cf8e-479d-bfc2-59514dff4b1c nodeName:}" failed. No retries permitted until 2026-07-24 12:53:38.681360101 +0000 UTC m=+42.670678779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls") pod "dns-default-hp2hl" (UID: "8354fa4f-cf8e-479d-bfc2-59514dff4b1c") : secret "dns-default-metrics-tls" not found
Jul 24 12:53:34.681517 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:34.681402    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls podName:3fbe6497-426b-41d3-a613-a792e28d5d68 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:38.681385185 +0000 UTC m=+42.670703874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls") pod "image-registry-745ff49dc7-zm67b" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68") : secret "image-registry-tls" not found
Jul 24 12:53:34.774707 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:34.774673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" event={"ID":"8bb5fd7a-722e-4d5a-98da-135ee39ff1a3","Type":"ContainerStarted","Data":"fbf4fbbb0f5ae7725bba58ec41112aa11dc1c3bb6d9d1454a6e5dbaeeaa5e036"}
Jul 24 12:53:34.805130 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:34.805085    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ldqh9" podStartSLOduration=6.699256293 podStartE2EDuration="37.805069125s" totalImagesPullingTime="31.105812832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:34.802988679 +0000 UTC m=+38.792307374" watchObservedRunningTime="2026-07-24 12:53:34.805069125 +0000 UTC m=+38.794387820"
Jul 24 12:53:35.778826 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:35.778791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8mj9k" event={"ID":"c42f561d-917e-4912-9c0d-7c7adcbdfb86","Type":"ContainerStarted","Data":"e8eba07c8e4064a0f0e7dfcdfcf51043522906a985e3206cb768aafeba22b45e"}
Jul 24 12:53:35.798579 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:35.798523    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8mj9k" podStartSLOduration=3.3091092890000002 podStartE2EDuration="5.798508713s" totalImagesPullingTime="2.489399424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:35.797276145 +0000 UTC m=+39.786594842" watchObservedRunningTime="2026-07-24 12:53:35.798508713 +0000 UTC m=+39.787827407"
Jul 24 12:53:36.171356 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.171318    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bmg69"]
Jul 24 12:53:36.187215 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.187191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.189478 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.189450    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 12:53:36.190101 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.190063    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 12:53:36.190101 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.190092    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 12:53:36.190215 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.190104    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 12:53:36.190604 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.190588    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ttjr9\""
Jul 24 12:53:36.192609 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.192583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmg69"]
Jul 24 12:53:36.292505 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.292470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.292505 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.292505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khww4\" (UniqueName: \"kubernetes.io/projected/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-api-access-khww4\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.292678 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.292533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.292678 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.292607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-data-volume\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.292678 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.292664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-crio-socket\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.319182 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.319154    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r"]
Jul 24 12:53:36.341193 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.341166    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r"]
Jul 24 12:53:36.341295 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.341273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r"
Jul 24 12:53:36.343903 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.343882    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 24 12:53:36.344062 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.343888    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-62bq9\""
Jul 24 12:53:36.344321 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.344304    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 24 12:53:36.393718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.393689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.393718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.393719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khww4\" (UniqueName: \"kubernetes.io/projected/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-api-access-khww4\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.393888 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.393748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.393888 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.393795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-data-volume\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.393888 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:36.393840    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:36.393888 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.393857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-crio-socket\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.394105 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:36.393894    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:36.893875835 +0000 UTC m=+40.883194526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:36.394105 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:36.393908    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:53:36.394105 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:36.393988    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:36.893970721 +0000 UTC m=+40.883289411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : configmap "kube-rbac-proxy" not found
Jul 24 12:53:36.394105 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.394089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-crio-socket\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.394270 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.394186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-data-volume\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.404481 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.404459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khww4\" (UniqueName: \"kubernetes.io/projected/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-api-access-khww4\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.495227 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.495152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxt6\" (UniqueName: \"kubernetes.io/projected/e336ec07-ea3f-45ac-9502-d55d3803b42c-kube-api-access-ngxt6\") pod \"migrator-6865c9967-crg9r\" (UID: \"e336ec07-ea3f-45ac-9502-d55d3803b42c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r"
Jul 24 12:53:36.595786 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.595751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxt6\" (UniqueName: \"kubernetes.io/projected/e336ec07-ea3f-45ac-9502-d55d3803b42c-kube-api-access-ngxt6\") pod \"migrator-6865c9967-crg9r\" (UID: \"e336ec07-ea3f-45ac-9502-d55d3803b42c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r"
Jul 24 12:53:36.610383 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.610360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxt6\" (UniqueName: \"kubernetes.io/projected/e336ec07-ea3f-45ac-9502-d55d3803b42c-kube-api-access-ngxt6\") pod \"migrator-6865c9967-crg9r\" (UID: \"e336ec07-ea3f-45ac-9502-d55d3803b42c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r"
Jul 24 12:53:36.650277 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.650245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r"
Jul 24 12:53:36.806647 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.806615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r"]
Jul 24 12:53:36.809451 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:53:36.809423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode336ec07_ea3f_45ac_9502_d55d3803b42c.slice/crio-4c9f7b74bf1108ed70098c9d2941aeb71a1dc10d903e3d44e67fb0c033838454 WatchSource:0}: Error finding container 4c9f7b74bf1108ed70098c9d2941aeb71a1dc10d903e3d44e67fb0c033838454: Status 404 returned error can't find the container with id 4c9f7b74bf1108ed70098c9d2941aeb71a1dc10d903e3d44e67fb0c033838454
Jul 24 12:53:36.898799 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.898763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.898960 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:36.898816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:36.898960 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:36.898890    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:53:36.898960 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:36.898901    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:36.899086 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:36.898963    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:37.898949644 +0000 UTC m=+41.888268318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : configmap "kube-rbac-proxy" not found
Jul 24 12:53:36.899086 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:36.898976    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:37.898970204 +0000 UTC m=+41.888288878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:37.767788 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:37.767761    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qf5nm_56249eea-b34a-4073-968c-a9e93e64e999/dns-node-resolver/0.log"
Jul 24 12:53:37.785174 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:37.785141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r" event={"ID":"e336ec07-ea3f-45ac-9502-d55d3803b42c","Type":"ContainerStarted","Data":"4c9f7b74bf1108ed70098c9d2941aeb71a1dc10d903e3d44e67fb0c033838454"}
Jul 24 12:53:37.906896 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:37.906859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:37.906896 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:37.906903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:37.907285 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:37.907025    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:53:37.907285 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:37.907026    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:37.907285 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:37.907084    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:39.907069119 +0000 UTC m=+43.896387797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:37.907285 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:37.907097    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:39.907090956 +0000 UTC m=+43.896409631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : configmap "kube-rbac-proxy" not found
Jul 24 12:53:38.714323 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:38.714291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:38.714323 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:38.714324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:38.714551 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:38.714422    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:53:38.714551 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:38.714428    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:53:38.714551 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:38.714445    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-745ff49dc7-zm67b: secret "image-registry-tls" not found
Jul 24 12:53:38.714551 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:38.714478    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls podName:8354fa4f-cf8e-479d-bfc2-59514dff4b1c nodeName:}" failed. No retries permitted until 2026-07-24 12:53:46.714460817 +0000 UTC m=+50.703779491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls") pod "dns-default-hp2hl" (UID: "8354fa4f-cf8e-479d-bfc2-59514dff4b1c") : secret "dns-default-metrics-tls" not found
Jul 24 12:53:38.714551 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:38.714504    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls podName:3fbe6497-426b-41d3-a613-a792e28d5d68 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:46.714492368 +0000 UTC m=+50.703811045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls") pod "image-registry-745ff49dc7-zm67b" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68") : secret "image-registry-tls" not found
Jul 24 12:53:38.788682 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:38.788650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r" event={"ID":"e336ec07-ea3f-45ac-9502-d55d3803b42c","Type":"ContainerStarted","Data":"a2363234108bec3ef527c56bb07bac079ca9aa875950cd0664ba9226e5e08aa5"}
Jul 24 12:53:38.788832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:38.788699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r" event={"ID":"e336ec07-ea3f-45ac-9502-d55d3803b42c","Type":"ContainerStarted","Data":"2e729d61db8ea76f90074d7bd4a17cd587a3208ee48282ef6fdf9a4268942580"}
Jul 24 12:53:38.809234 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:38.809182    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-crg9r" podStartSLOduration=1.347758009 podStartE2EDuration="2.809168409s" totalImagesPullingTime="1.4614104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:38.808483742 +0000 UTC m=+42.797802438" watchObservedRunningTime="2026-07-24 12:53:38.809168409 +0000 UTC m=+42.798487105"
Jul 24 12:53:38.969013 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:38.968913    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zl565_a39d5644-41ba-439f-87e2-72f2dfae3422/node-ca/0.log"
Jul 24 12:53:39.377350 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:39.377269    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8mj9k_c42f561d-917e-4912-9c0d-7c7adcbdfb86/serve-healthcheck-canary/0.log"
Jul 24 12:53:39.922116 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:39.922074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:39.922291 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:39.922190    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:53:39.922291 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:39.922214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:39.922291 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:39.922256    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:43.922241417 +0000 UTC m=+47.911560091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : configmap "kube-rbac-proxy" not found
Jul 24 12:53:39.922460 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:39.922321    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:39.922460 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:39.922380    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:43.922363683 +0000 UTC m=+47.911682381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:43.951076 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:43.951032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:43.951530 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:43.951084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:43.951530 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:43.951189    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:43.951530 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:43.951260    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls podName:e2ff83e6-089c-4920-ae44-4f4c8fbc5737 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:51.951242712 +0000 UTC m=+55.940561390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmg69" (UID: "e2ff83e6-089c-4920-ae44-4f4c8fbc5737") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:53:43.951656 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:43.951636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:46.769898 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:46.769864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:53:46.769898 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:46.769896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:53:46.770316 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:46.770004    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:53:46.770316 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:46.770009    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:53:46.770316 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:46.770026    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-745ff49dc7-zm67b: secret "image-registry-tls" not found
Jul 24 12:53:46.770316 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:46.770056    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls podName:8354fa4f-cf8e-479d-bfc2-59514dff4b1c nodeName:}" failed. No retries permitted until 2026-07-24 12:54:02.770043691 +0000 UTC m=+66.759362365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls") pod "dns-default-hp2hl" (UID: "8354fa4f-cf8e-479d-bfc2-59514dff4b1c") : secret "dns-default-metrics-tls" not found
Jul 24 12:53:46.770316 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:53:46.770075    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls podName:3fbe6497-426b-41d3-a613-a792e28d5d68 nodeName:}" failed. No retries permitted until 2026-07-24 12:54:02.770063454 +0000 UTC m=+66.759382152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls") pod "image-registry-745ff49dc7-zm67b" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68") : secret "image-registry-tls" not found
Jul 24 12:53:52.016915 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:52.016858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:52.019360 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:52.019335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e2ff83e6-089c-4920-ae44-4f4c8fbc5737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmg69\" (UID: \"e2ff83e6-089c-4920-ae44-4f4c8fbc5737\") " pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:52.096371 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:52.096340    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmg69"
Jul 24 12:53:52.228676 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:52.228650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmg69"]
Jul 24 12:53:52.230869 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:53:52.230843    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ff83e6_089c_4920_ae44_4f4c8fbc5737.slice/crio-acdd0893383b0135a31e0fc6a493999a978fe9dccdb25e410dff12b819822221 WatchSource:0}: Error finding container acdd0893383b0135a31e0fc6a493999a978fe9dccdb25e410dff12b819822221: Status 404 returned error can't find the container with id acdd0893383b0135a31e0fc6a493999a978fe9dccdb25e410dff12b819822221
Jul 24 12:53:52.825006 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:52.824965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmg69" event={"ID":"e2ff83e6-089c-4920-ae44-4f4c8fbc5737","Type":"ContainerStarted","Data":"acdd0893383b0135a31e0fc6a493999a978fe9dccdb25e410dff12b819822221"}
Jul 24 12:53:56.753001 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:56.752974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x2pkb"
Jul 24 12:53:57.841325 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:53:57.841243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmg69" event={"ID":"e2ff83e6-089c-4920-ae44-4f4c8fbc5737","Type":"ContainerStarted","Data":"038181fbf175ea405364572ca09faee01eec45d3092d9fdfa800da73750e577c"}
Jul 24 12:54:00.854282 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:00.854250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmg69" event={"ID":"e2ff83e6-089c-4920-ae44-4f4c8fbc5737","Type":"ContainerStarted","Data":"c17acc3ed8fb39dda140881c5029634321758a62ec4e72639250afcd24796630"}
Jul 24 12:54:01.294457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.294411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:54:01.294626 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.294471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:54:01.297642 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.297611    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 12:54:01.297779 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.297614    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 12:54:01.307258 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.307236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e1bc47e-ead0-460e-92ba-e57dfafcde04-metrics-certs\") pod \"network-metrics-daemon-8nqcd\" (UID: \"2e1bc47e-ead0-460e-92ba-e57dfafcde04\") " pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:54:01.307357 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.307264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6956353c-c03e-4a3c-8a33-a098f4c11e35-original-pull-secret\") pod \"global-pull-secret-syncer-shckb\" (UID: \"6956353c-c03e-4a3c-8a33-a098f4c11e35\") " pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:54:01.395623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.395594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:54:01.398774 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.398753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 12:54:01.409267 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.409241    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 12:54:01.419468 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.419444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrsn\" (UniqueName: \"kubernetes.io/projected/d90b3a00-d6a3-457f-a8bb-5f2bde5754b1-kube-api-access-nlrsn\") pod \"network-check-target-nn8v9\" (UID: \"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1\") " pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:54:01.572087 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.572005    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lv5pf\""
Jul 24 12:54:01.578081 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.578055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-txwsv\""
Jul 24 12:54:01.579074 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.579054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:54:01.580671 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.580646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shckb"
Jul 24 12:54:01.586157 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.586135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nqcd"
Jul 24 12:54:01.743316 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.743251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nn8v9"]
Jul 24 12:54:01.745835 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:01.745801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90b3a00_d6a3_457f_a8bb_5f2bde5754b1.slice/crio-70a82bd2969ae93586b36db29f7e7869299e6876b812c8cd9688692f78a8b9ba WatchSource:0}: Error finding container 70a82bd2969ae93586b36db29f7e7869299e6876b812c8cd9688692f78a8b9ba: Status 404 returned error can't find the container with id 70a82bd2969ae93586b36db29f7e7869299e6876b812c8cd9688692f78a8b9ba
Jul 24 12:54:01.760448 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.760425    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8nqcd"]
Jul 24 12:54:01.762840 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.762818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-shckb"]
Jul 24 12:54:01.763237 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:01.763217    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1bc47e_ead0_460e_92ba_e57dfafcde04.slice/crio-10e7639d9086b2d51221a674ac0a1543ac3a3320187bca0e21e2947f58de7d7b WatchSource:0}: Error finding container 10e7639d9086b2d51221a674ac0a1543ac3a3320187bca0e21e2947f58de7d7b: Status 404 returned error can't find the container with id 10e7639d9086b2d51221a674ac0a1543ac3a3320187bca0e21e2947f58de7d7b
Jul 24 12:54:01.766424 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:01.766293    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6956353c_c03e_4a3c_8a33_a098f4c11e35.slice/crio-81813d468d6a97a42ddf579f663c7f2055f299630baa283c94d3193d3fbd8e8f WatchSource:0}: Error finding container 81813d468d6a97a42ddf579f663c7f2055f299630baa283c94d3193d3fbd8e8f: Status 404 returned error can't find the container with id 81813d468d6a97a42ddf579f663c7f2055f299630baa283c94d3193d3fbd8e8f
Jul 24 12:54:01.858658 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.858575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-shckb" event={"ID":"6956353c-c03e-4a3c-8a33-a098f4c11e35","Type":"ContainerStarted","Data":"81813d468d6a97a42ddf579f663c7f2055f299630baa283c94d3193d3fbd8e8f"}
Jul 24 12:54:01.859819 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.859789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nqcd" event={"ID":"2e1bc47e-ead0-460e-92ba-e57dfafcde04","Type":"ContainerStarted","Data":"10e7639d9086b2d51221a674ac0a1543ac3a3320187bca0e21e2947f58de7d7b"}
Jul 24 12:54:01.860999 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:01.860972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nn8v9" event={"ID":"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1","Type":"ContainerStarted","Data":"70a82bd2969ae93586b36db29f7e7869299e6876b812c8cd9688692f78a8b9ba"}
Jul 24 12:54:02.807407 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:02.807372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:54:02.807407 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:02.807412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:54:02.810295 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:02.810271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8354fa4f-cf8e-479d-bfc2-59514dff4b1c-metrics-tls\") pod \"dns-default-hp2hl\" (UID: \"8354fa4f-cf8e-479d-bfc2-59514dff4b1c\") " pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:54:02.810407 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:02.810325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"image-registry-745ff49dc7-zm67b\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") " pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:54:03.018227 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.018188    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"]
Jul 24 12:54:03.037120 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.037089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"]
Jul 24 12:54:03.037275 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.037215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"
Jul 24 12:54:03.041128 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.041098    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-745ff49dc7-zm67b"]
Jul 24 12:54:03.041859 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.041835    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 12:54:03.041994 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.041870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 12:54:03.042116 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.042098    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 12:54:03.042186 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.042135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 12:54:03.042399 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.042382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4tlrl\""
Jul 24 12:54:03.044266 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.044244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zfr64\""
Jul 24 12:54:03.051843 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.051819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:54:03.081793 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.081592    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7xh6d\""
Jul 24 12:54:03.088736 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.088710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:54:03.110280 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.110239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45e0ad25-3162-4774-9887-15487b5e03bc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz\" (UID: \"45e0ad25-3162-4774-9887-15487b5e03bc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"
Jul 24 12:54:03.111236 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.111202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krkp4\" (UniqueName: \"kubernetes.io/projected/45e0ad25-3162-4774-9887-15487b5e03bc-kube-api-access-krkp4\") pod \"managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz\" (UID: \"45e0ad25-3162-4774-9887-15487b5e03bc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"
Jul 24 12:54:03.112019 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.111998    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-twrff"]
Jul 24 12:54:03.136222 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.135165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-wrsk5"]
Jul 24 12:54:03.136222 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.135896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff"
Jul 24 12:54:03.140660 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.140617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dtz6j\""
Jul 24 12:54:03.141130 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.140951    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 12:54:03.142101 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.142076    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 12:54:03.150088 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.150062    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9d76c648c-5m9xs"]
Jul 24 12:54:03.150212 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.150195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-wrsk5"
Jul 24 12:54:03.154911 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.154887    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 12:54:03.155260 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.155242    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xxrrp\""
Jul 24 12:54:03.155346 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.155289    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 12:54:03.166165 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.166137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-twrff"]
Jul 24 12:54:03.166329 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.166187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-wrsk5"]
Jul 24 12:54:03.166329 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.166211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9d76c648c-5m9xs"]
Jul 24 12:54:03.166329 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.166311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.212406 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7cae0063-69fe-4e37-a7cd-cbf5be03f541-image-registry-private-configuration\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.212598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7cae0063-69fe-4e37-a7cd-cbf5be03f541-ca-trust-extracted\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.212598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f579dc6a-bffa-499c-8540-0e952bb62a06-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-twrff\" (UID: \"f579dc6a-bffa-499c-8540-0e952bb62a06\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff"
Jul 24 12:54:03.212598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45e0ad25-3162-4774-9887-15487b5e03bc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz\" (UID: \"45e0ad25-3162-4774-9887-15487b5e03bc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"
Jul 24 12:54:03.212598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f579dc6a-bffa-499c-8540-0e952bb62a06-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-twrff\" (UID: \"f579dc6a-bffa-499c-8540-0e952bb62a06\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff"
Jul 24 12:54:03.212598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7cae0063-69fe-4e37-a7cd-cbf5be03f541-installation-pull-secrets\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.212598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7cae0063-69fe-4e37-a7cd-cbf5be03f541-registry-certificates\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.212598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krkp4\" (UniqueName: \"kubernetes.io/projected/45e0ad25-3162-4774-9887-15487b5e03bc-kube-api-access-krkp4\") pod \"managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz\" (UID: \"45e0ad25-3162-4774-9887-15487b5e03bc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"
Jul 24 12:54:03.213020 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-bound-sa-token\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.213020 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7spl\" (UniqueName: \"kubernetes.io/projected/ab5c680f-75a8-4c64-a9cc-70ce761974dd-kube-api-access-w7spl\") pod \"downloads-7c97dc4555-wrsk5\" (UID: \"ab5c680f-75a8-4c64-a9cc-70ce761974dd\") " pod="openshift-console/downloads-7c97dc4555-wrsk5"
Jul 24 12:54:03.213020 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-registry-tls\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.213020 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cae0063-69fe-4e37-a7cd-cbf5be03f541-trusted-ca\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.213020 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.212782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr67w\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-kube-api-access-nr67w\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.227717 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.227685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45e0ad25-3162-4774-9887-15487b5e03bc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz\" (UID: \"45e0ad25-3162-4774-9887-15487b5e03bc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"
Jul 24 12:54:03.229461 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.229406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krkp4\" (UniqueName: \"kubernetes.io/projected/45e0ad25-3162-4774-9887-15487b5e03bc-kube-api-access-krkp4\") pod \"managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz\" (UID: \"45e0ad25-3162-4774-9887-15487b5e03bc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"
Jul 24 12:54:03.314299 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-bound-sa-token\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.314299 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7spl\" (UniqueName: \"kubernetes.io/projected/ab5c680f-75a8-4c64-a9cc-70ce761974dd-kube-api-access-w7spl\") pod \"downloads-7c97dc4555-wrsk5\" (UID: \"ab5c680f-75a8-4c64-a9cc-70ce761974dd\") " pod="openshift-console/downloads-7c97dc4555-wrsk5"
Jul 24 12:54:03.314532 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-registry-tls\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.314532 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cae0063-69fe-4e37-a7cd-cbf5be03f541-trusted-ca\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.314532 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr67w\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-kube-api-access-nr67w\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.314532 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7cae0063-69fe-4e37-a7cd-cbf5be03f541-image-registry-private-configuration\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.314532 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7cae0063-69fe-4e37-a7cd-cbf5be03f541-ca-trust-extracted\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.314532 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f579dc6a-bffa-499c-8540-0e952bb62a06-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-twrff\" (UID: \"f579dc6a-bffa-499c-8540-0e952bb62a06\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff"
Jul 24 12:54:03.314532 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f579dc6a-bffa-499c-8540-0e952bb62a06-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-twrff\" (UID: \"f579dc6a-bffa-499c-8540-0e952bb62a06\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff"
Jul 24 12:54:03.314785 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7cae0063-69fe-4e37-a7cd-cbf5be03f541-installation-pull-secrets\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.314785 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.314587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7cae0063-69fe-4e37-a7cd-cbf5be03f541-registry-certificates\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.317184 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.315600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f579dc6a-bffa-499c-8540-0e952bb62a06-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-twrff\" (UID: \"f579dc6a-bffa-499c-8540-0e952bb62a06\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff"
Jul 24 12:54:03.317184 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.315774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7cae0063-69fe-4e37-a7cd-cbf5be03f541-registry-certificates\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.317184 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.315875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7cae0063-69fe-4e37-a7cd-cbf5be03f541-ca-trust-extracted\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.317184 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.316714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cae0063-69fe-4e37-a7cd-cbf5be03f541-trusted-ca\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.318507 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.318473    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f579dc6a-bffa-499c-8540-0e952bb62a06-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-twrff\" (UID: \"f579dc6a-bffa-499c-8540-0e952bb62a06\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff"
Jul 24 12:54:03.319757 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.319717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-registry-tls\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.320214 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.320192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7cae0063-69fe-4e37-a7cd-cbf5be03f541-installation-pull-secrets\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.322294 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.322268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7cae0063-69fe-4e37-a7cd-cbf5be03f541-image-registry-private-configuration\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.329683 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.329624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7spl\" (UniqueName: \"kubernetes.io/projected/ab5c680f-75a8-4c64-a9cc-70ce761974dd-kube-api-access-w7spl\") pod \"downloads-7c97dc4555-wrsk5\" (UID: \"ab5c680f-75a8-4c64-a9cc-70ce761974dd\") " pod="openshift-console/downloads-7c97dc4555-wrsk5"
Jul 24 12:54:03.331275 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.331215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr67w\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-kube-api-access-nr67w\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.332694 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.332673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cae0063-69fe-4e37-a7cd-cbf5be03f541-bound-sa-token\") pod \"image-registry-9d76c648c-5m9xs\" (UID: \"7cae0063-69fe-4e37-a7cd-cbf5be03f541\") " pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:03.357810 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.357756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"
Jul 24 12:54:03.447947 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.447899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff"
Jul 24 12:54:03.461312 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.461240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-wrsk5"
Jul 24 12:54:03.478323 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:03.478291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:06.210269 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.210235    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"]
Jul 24 12:54:06.212606 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.212582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"
Jul 24 12:54:06.215336 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.215314    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 12:54:06.215433 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.215321    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-s9brj\""
Jul 24 12:54:06.224819 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.224792    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"]
Jul 24 12:54:06.337598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.337558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4caef4b-fd87-4760-b8d8-3957496cc44e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-vwcg8\" (UID: \"f4caef4b-fd87-4760-b8d8-3957496cc44e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"
Jul 24 12:54:06.438018 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.437981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4caef4b-fd87-4760-b8d8-3957496cc44e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-vwcg8\" (UID: \"f4caef4b-fd87-4760-b8d8-3957496cc44e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"
Jul 24 12:54:06.441001 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.440974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4caef4b-fd87-4760-b8d8-3957496cc44e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-vwcg8\" (UID: \"f4caef4b-fd87-4760-b8d8-3957496cc44e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"
Jul 24 12:54:06.522276 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:06.522181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"
Jul 24 12:54:07.766368 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.766144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9d76c648c-5m9xs"]
Jul 24 12:54:07.786102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.784455    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-twrff"]
Jul 24 12:54:07.796620 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.796204    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"]
Jul 24 12:54:07.810614 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:07.810570    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4caef4b_fd87_4760_b8d8_3957496cc44e.slice/crio-f7ca94fc96467961821698d02ed55d2868daaabe678273045b7b3cef2e0cfd0a WatchSource:0}: Error finding container f7ca94fc96467961821698d02ed55d2868daaabe678273045b7b3cef2e0cfd0a: Status 404 returned error can't find the container with id f7ca94fc96467961821698d02ed55d2868daaabe678273045b7b3cef2e0cfd0a
Jul 24 12:54:07.816886 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.816724    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-745ff49dc7-zm67b"]
Jul 24 12:54:07.823284 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:07.823246    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbe6497_426b_41d3_a613_a792e28d5d68.slice/crio-cddff5b6aa67fa5ce166fb4c032ae51485e7bfd5c4603fd0739151531eb227d3 WatchSource:0}: Error finding container cddff5b6aa67fa5ce166fb4c032ae51485e7bfd5c4603fd0739151531eb227d3: Status 404 returned error can't find the container with id cddff5b6aa67fa5ce166fb4c032ae51485e7bfd5c4603fd0739151531eb227d3
Jul 24 12:54:07.827758 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.826820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hp2hl"]
Jul 24 12:54:07.850267 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.850193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-wrsk5"]
Jul 24 12:54:07.857947 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:07.857901    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5c680f_75a8_4c64_a9cc_70ce761974dd.slice/crio-1190286f33a258154ddf86f1f62bb24860d8f307eab2e6cad0bf2a0718cbe60a WatchSource:0}: Error finding container 1190286f33a258154ddf86f1f62bb24860d8f307eab2e6cad0bf2a0718cbe60a: Status 404 returned error can't find the container with id 1190286f33a258154ddf86f1f62bb24860d8f307eab2e6cad0bf2a0718cbe60a
Jul 24 12:54:07.885355 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.885293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-shckb" event={"ID":"6956353c-c03e-4a3c-8a33-a098f4c11e35","Type":"ContainerStarted","Data":"ab7ebf5b5be28d6c539ea24e298e96bc8be9ce8fed07fb76706f0918e4d61754"}
Jul 24 12:54:07.888483 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.888453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmg69" event={"ID":"e2ff83e6-089c-4920-ae44-4f4c8fbc5737","Type":"ContainerStarted","Data":"88585718cdb1e39eef10dc24af2c6525fd324c53865326ace3ef0884b77b0f9f"}
Jul 24 12:54:07.892727 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.892441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nqcd" event={"ID":"2e1bc47e-ead0-460e-92ba-e57dfafcde04","Type":"ContainerStarted","Data":"897a8148f80019e1de32896991ba18cac74d5bbc3a683ce90cd7319607faa386"}
Jul 24 12:54:07.894117 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.894094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-wrsk5" event={"ID":"ab5c680f-75a8-4c64-a9cc-70ce761974dd","Type":"ContainerStarted","Data":"1190286f33a258154ddf86f1f62bb24860d8f307eab2e6cad0bf2a0718cbe60a"}
Jul 24 12:54:07.895613 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.895555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8" event={"ID":"f4caef4b-fd87-4760-b8d8-3957496cc44e","Type":"ContainerStarted","Data":"f7ca94fc96467961821698d02ed55d2868daaabe678273045b7b3cef2e0cfd0a"}
Jul 24 12:54:07.896940 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.896897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff" event={"ID":"f579dc6a-bffa-499c-8540-0e952bb62a06","Type":"ContainerStarted","Data":"7cf9804bdf7eda221de01a17ba5e8e003e4843a808bb598ab8bdb110aff52999"}
Jul 24 12:54:07.898771 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.898747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b" event={"ID":"3fbe6497-426b-41d3-a613-a792e28d5d68","Type":"ContainerStarted","Data":"cddff5b6aa67fa5ce166fb4c032ae51485e7bfd5c4603fd0739151531eb227d3"}
Jul 24 12:54:07.898864 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.898829    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:54:07.901225 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.901198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nn8v9" event={"ID":"d90b3a00-d6a3-457f-a8bb-5f2bde5754b1","Type":"ContainerStarted","Data":"d9e34ecd7ca166706e46fa386f60aef7344521eb30a464eb55248b5d84d1358c"}
Jul 24 12:54:07.901694 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.901675    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:54:07.903604 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.903339    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-shckb" podStartSLOduration=65.168206466 podStartE2EDuration="1m10.903322078s" totalImagesPullingTime="5.735115612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:07.90212063 +0000 UTC m=+71.891439327" watchObservedRunningTime="2026-07-24 12:54:07.903322078 +0000 UTC m=+71.892640766"
Jul 24 12:54:07.905291 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.905234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hp2hl" event={"ID":"8354fa4f-cf8e-479d-bfc2-59514dff4b1c","Type":"ContainerStarted","Data":"e3d1afb9a1498bbc47018eac90b7bcf9e8f400b19b101fd0bc0cd354cfc70439"}
Jul 24 12:54:07.907585 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.907567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9d76c648c-5m9xs" event={"ID":"7cae0063-69fe-4e37-a7cd-cbf5be03f541","Type":"ContainerStarted","Data":"d704c373706401849191495c0b9b752b86eb48dd20692a34ddf0858156e5c0dd"}
Jul 24 12:54:07.908102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.908077    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:07.919698 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.919649    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nn8v9" podStartSLOduration=65.175778926 podStartE2EDuration="1m10.919637955s" totalImagesPullingTime="5.743859029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:07.918805207 +0000 UTC m=+71.908123903" watchObservedRunningTime="2026-07-24 12:54:07.919637955 +0000 UTC m=+71.908956651"
Jul 24 12:54:07.944600 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.944319    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b" podStartSLOduration=52.944296859 podStartE2EDuration="52.944296859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:07.943732169 +0000 UTC m=+71.933050867" watchObservedRunningTime="2026-07-24 12:54:07.944296859 +0000 UTC m=+71.933615556"
Jul 24 12:54:07.985661 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.985552    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bmg69" podStartSLOduration=16.824122215 podStartE2EDuration="31.985530351s" totalImagesPullingTime="15.161408136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:07.984941723 +0000 UTC m=+71.974260421" watchObservedRunningTime="2026-07-24 12:54:07.985530351 +0000 UTC m=+71.974849050"
Jul 24 12:54:07.993799 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:07.993759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz"]
Jul 24 12:54:07.996816 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:07.996782    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45e0ad25_3162_4774_9887_15487b5e03bc.slice/crio-52171e7de89497f308329bd0a7d2f38c62ab94e7af1cd7193725eeff477abaf0 WatchSource:0}: Error finding container 52171e7de89497f308329bd0a7d2f38c62ab94e7af1cd7193725eeff477abaf0: Status 404 returned error can't find the container with id 52171e7de89497f308329bd0a7d2f38c62ab94e7af1cd7193725eeff477abaf0
Jul 24 12:54:08.022520 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:08.022462    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8nqcd" podStartSLOduration=65.300685745 podStartE2EDuration="1m11.02244295s" totalImagesPullingTime="5.721757205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:08.022120285 +0000 UTC m=+72.011438981" watchObservedRunningTime="2026-07-24 12:54:08.02244295 +0000 UTC m=+72.011761646"
Jul 24 12:54:08.045286 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:08.045219    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9d76c648c-5m9xs" podStartSLOduration=5.045194902 podStartE2EDuration="5.045194902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:08.044143573 +0000 UTC m=+72.033462268" watchObservedRunningTime="2026-07-24 12:54:08.045194902 +0000 UTC m=+72.034513601"
Jul 24 12:54:08.923540 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:08.923487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz" event={"ID":"45e0ad25-3162-4774-9887-15487b5e03bc","Type":"ContainerStarted","Data":"52171e7de89497f308329bd0a7d2f38c62ab94e7af1cd7193725eeff477abaf0"}
Jul 24 12:54:08.926451 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:08.926343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nqcd" event={"ID":"2e1bc47e-ead0-460e-92ba-e57dfafcde04","Type":"ContainerStarted","Data":"344e5ec139fda5c7177ce75353f4fbd099e77e6fe59e23247e1000d82905eab7"}
Jul 24 12:54:08.928357 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:08.928322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b" event={"ID":"3fbe6497-426b-41d3-a613-a792e28d5d68","Type":"ContainerStarted","Data":"bc0dffacbec78a34e606a61af60023b16edf3bd6bbf1c5370926919cfdcc136e"}
Jul 24 12:54:08.931188 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:08.931159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9d76c648c-5m9xs" event={"ID":"7cae0063-69fe-4e37-a7cd-cbf5be03f541","Type":"ContainerStarted","Data":"b0ea40af95eff3ffd90dc17777f59dcd455af932c45eb74d9896222afd3ae1b8"}
Jul 24 12:54:10.940900 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:10.940861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8" event={"ID":"f4caef4b-fd87-4760-b8d8-3957496cc44e","Type":"ContainerStarted","Data":"db7214169a77bbd3d957fd59042151fcf84f2f3073094ffd0adcedfefabdea41"}
Jul 24 12:54:10.941396 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:10.941119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"
Jul 24 12:54:10.943197 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:10.942798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff" event={"ID":"f579dc6a-bffa-499c-8540-0e952bb62a06","Type":"ContainerStarted","Data":"53497f134fea735a22d9e5ad777567705c6a36aa60b1f307da0e2d15c58ebf3a"}
Jul 24 12:54:10.944964 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:10.944908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hp2hl" event={"ID":"8354fa4f-cf8e-479d-bfc2-59514dff4b1c","Type":"ContainerStarted","Data":"4d8b2311e3a856905bb732647840a8742d860d4c838ead4267fe15c1710086e8"}
Jul 24 12:54:10.945361 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:10.945338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:54:10.945468 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:10.945380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hp2hl" event={"ID":"8354fa4f-cf8e-479d-bfc2-59514dff4b1c","Type":"ContainerStarted","Data":"4d961440abe91ea4b4f8156fbe26fb99b28fd78645bbf98c4a69e3a7913448e8"}
Jul 24 12:54:10.946362 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:10.946343    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8"
Jul 24 12:54:10.974579 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:10.974528    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-vwcg8" podStartSLOduration=2.690847812 podStartE2EDuration="4.974506316s" totalImagesPullingTime="2.283658504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:10.958746408 +0000 UTC m=+74.948065104" watchObservedRunningTime="2026-07-24 12:54:10.974506316 +0000 UTC m=+74.963825031"
Jul 24 12:54:11.000551 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.000488    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-twrff" podStartSLOduration=5.712787341 podStartE2EDuration="8.000466568s" totalImagesPullingTime="2.287679227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:10.975828295 +0000 UTC m=+74.965146990" watchObservedRunningTime="2026-07-24 12:54:11.000466568 +0000 UTC m=+74.989785262"
Jul 24 12:54:11.001318 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.001283    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hp2hl" podStartSLOduration=38.690922392 podStartE2EDuration="41.001272204s" totalImagesPullingTime="2.310349812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:10.999562386 +0000 UTC m=+74.988881082" watchObservedRunningTime="2026-07-24 12:54:11.001272204 +0000 UTC m=+74.990590900"
Jul 24 12:54:11.290255 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.290180    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"]
Jul 24 12:54:11.294474 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.294446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.297135 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.297107    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 12:54:11.297411 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.297382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 12:54:11.297411 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.297398    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 12:54:11.297582 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.297466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 12:54:11.297669 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.297590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 12:54:11.298354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.298329    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wlsrj\""
Jul 24 12:54:11.303208 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.303180    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"]
Jul 24 12:54:11.388849 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.388809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.389037 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.388868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjp74\" (UniqueName: \"kubernetes.io/projected/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-kube-api-access-jjp74\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.389037 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.388904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.389037 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.388995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.489651 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.489609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjp74\" (UniqueName: \"kubernetes.io/projected/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-kube-api-access-jjp74\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.489839 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.489669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.489839 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.489711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.489839 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.489748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.490468 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.490411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.492382 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.492357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.492549 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.492528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.499349 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.499328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjp74\" (UniqueName: \"kubernetes.io/projected/a6a3a4eb-576a-48d5-97e8-06f233fdca9a-kube-api-access-jjp74\") pod \"prometheus-operator-84d69f794d-8k4kq\" (UID: \"a6a3a4eb-576a-48d5-97e8-06f233fdca9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.606306 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.606222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"
Jul 24 12:54:11.749643 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.749608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-8k4kq"]
Jul 24 12:54:11.753108 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:11.753077    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a3a4eb_576a_48d5_97e8_06f233fdca9a.slice/crio-2ae88c4ca99c8c7c4157c5efdec6a77650512c1eca82bee36ff62140ad0d28f5 WatchSource:0}: Error finding container 2ae88c4ca99c8c7c4157c5efdec6a77650512c1eca82bee36ff62140ad0d28f5: Status 404 returned error can't find the container with id 2ae88c4ca99c8c7c4157c5efdec6a77650512c1eca82bee36ff62140ad0d28f5
Jul 24 12:54:11.950832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:11.950795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq" event={"ID":"a6a3a4eb-576a-48d5-97e8-06f233fdca9a","Type":"ContainerStarted","Data":"2ae88c4ca99c8c7c4157c5efdec6a77650512c1eca82bee36ff62140ad0d28f5"}
Jul 24 12:54:12.054027 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.053991    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77697885cd-25q5t"]
Jul 24 12:54:12.060143 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.060115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.063251 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.063220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b8dbp\""
Jul 24 12:54:12.063491 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.063470    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 12:54:12.064469 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.064445    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 12:54:12.064550 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.064520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 12:54:12.064657 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.064641    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 12:54:12.064716 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.064703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 12:54:12.068854 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.068793    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77697885cd-25q5t"]
Jul 24 12:54:12.069124 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.069101    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 12:54:12.195500 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.195462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-console-config\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.195684 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.195507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-service-ca\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.195684 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.195594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-serving-cert\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.195684 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.195656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-oauth-config\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.195684 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.195673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-trusted-ca-bundle\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.195892 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.195701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-oauth-serving-cert\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.195892 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.195721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5p9\" (UniqueName: \"kubernetes.io/projected/b4f5e4eb-0640-4125-998b-4fb450044413-kube-api-access-6c5p9\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.296850 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.296762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-oauth-serving-cert\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.296850 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.296799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5p9\" (UniqueName: \"kubernetes.io/projected/b4f5e4eb-0640-4125-998b-4fb450044413-kube-api-access-6c5p9\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.296850 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.296844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-console-config\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.297183 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.296869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-service-ca\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.297183 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.296911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-serving-cert\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.297183 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.296980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-oauth-config\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.297183 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.297001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-trusted-ca-bundle\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.297538 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.297510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-oauth-serving-cert\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.297809 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.297790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-trusted-ca-bundle\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.297809 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.297802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-service-ca\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.298102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.298077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-console-config\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.300507 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.300483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-oauth-config\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.300609 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.300575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-serving-cert\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.305889 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.305868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5p9\" (UniqueName: \"kubernetes.io/projected/b4f5e4eb-0640-4125-998b-4fb450044413-kube-api-access-6c5p9\") pod \"console-77697885cd-25q5t\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") " pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.372205 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.372164    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:12.527734 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.527694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77697885cd-25q5t"]
Jul 24 12:54:12.532035 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:12.532001    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f5e4eb_0640_4125_998b_4fb450044413.slice/crio-d6fa94903c0600bcdd124d3c6c85888df6c5b2b799055396609e22cb9cc22490 WatchSource:0}: Error finding container d6fa94903c0600bcdd124d3c6c85888df6c5b2b799055396609e22cb9cc22490: Status 404 returned error can't find the container with id d6fa94903c0600bcdd124d3c6c85888df6c5b2b799055396609e22cb9cc22490
Jul 24 12:54:12.956567 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:12.956531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77697885cd-25q5t" event={"ID":"b4f5e4eb-0640-4125-998b-4fb450044413","Type":"ContainerStarted","Data":"d6fa94903c0600bcdd124d3c6c85888df6c5b2b799055396609e22cb9cc22490"}
Jul 24 12:54:13.963244 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:13.963202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq" event={"ID":"a6a3a4eb-576a-48d5-97e8-06f233fdca9a","Type":"ContainerStarted","Data":"21afa373e3a4f932be624eb2356aa5acc3ecf34ee33aa1c0b10137d7692e6e08"}
Jul 24 12:54:13.963690 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:13.963274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq" event={"ID":"a6a3a4eb-576a-48d5-97e8-06f233fdca9a","Type":"ContainerStarted","Data":"d2e050d28c365530aeb70053018069cc9e0702987f6e3b21a8b34083bcf54bee"}
Jul 24 12:54:13.982340 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:13.982271    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-8k4kq" podStartSLOduration=1.54003866 podStartE2EDuration="2.982249846s" totalImagesPullingTime="1.442211186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:13.981067705 +0000 UTC m=+77.970386401" watchObservedRunningTime="2026-07-24 12:54:13.982249846 +0000 UTC m=+77.971568543"
Jul 24 12:54:15.697765 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.697729    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qz7x7"]
Jul 24 12:54:15.705312 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.705280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.708352 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.707868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 12:54:15.708352 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.708154    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 12:54:15.708352 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.708169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9srnh\""
Jul 24 12:54:15.708610 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.708370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 12:54:15.828849 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.828816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-wtmp\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.828974 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.828899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.828974 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.828963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.829094 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.829008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-root\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.829094 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.829057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-textfile\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.829188 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.829106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-tls\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.829188 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.829141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-metrics-client-ca\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.829188 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.829164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z697x\" (UniqueName: \"kubernetes.io/projected/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-kube-api-access-z697x\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.829334 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.829196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-sys\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-wtmp\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-root\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-textfile\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-tls\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-metrics-client-ca\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-root\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z697x\" (UniqueName: \"kubernetes.io/projected/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-kube-api-access-z697x\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-wtmp\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-sys\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.929989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-sys\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930054 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.930036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-textfile\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930779 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.930362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.930779 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.930757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-metrics-client-ca\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.932571 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.932524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.933147 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.933124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-node-exporter-tls\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.939739 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.939696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z697x\" (UniqueName: \"kubernetes.io/projected/7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9-kube-api-access-z697x\") pod \"node-exporter-qz7x7\" (UID: \"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9\") " pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:15.973901 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.973803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77697885cd-25q5t" event={"ID":"b4f5e4eb-0640-4125-998b-4fb450044413","Type":"ContainerStarted","Data":"ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe"}
Jul 24 12:54:15.993038 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:15.992983    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77697885cd-25q5t" podStartSLOduration=0.734343886 podStartE2EDuration="3.992961687s" totalImagesPullingTime="3.258617801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:15.991739703 +0000 UTC m=+79.981058411" watchObservedRunningTime="2026-07-24 12:54:15.992961687 +0000 UTC m=+79.982280385"
Jul 24 12:54:16.017146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.017111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qz7x7"
Jul 24 12:54:16.030482 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:16.030452    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7788ab9f_b4bd_4b7b_87f4_a1b6b6f8e9b9.slice/crio-5acaa573a00ea9f5b7df91c674a7f5de45f3f2b5a9ffd0c56c524ceb7009f0a0 WatchSource:0}: Error finding container 5acaa573a00ea9f5b7df91c674a7f5de45f3f2b5a9ffd0c56c524ceb7009f0a0: Status 404 returned error can't find the container with id 5acaa573a00ea9f5b7df91c674a7f5de45f3f2b5a9ffd0c56c524ceb7009f0a0
Jul 24 12:54:16.805240 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.805202    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:54:16.812021 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.810281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.814315 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.813181    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 12:54:16.814315 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.813422    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jp87l\""
Jul 24 12:54:16.814315 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.813770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 12:54:16.814315 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.813972    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 12:54:16.814315 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.814111    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 12:54:16.814315 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.814151    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 12:54:16.814315 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.814211    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 12:54:16.814714 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.814362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 12:54:16.814768 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.814726    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 12:54:16.814980 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.814959    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 12:54:16.825016 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.824896    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:54:16.938277 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-out\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938277 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938494 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938494 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938494 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938494 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938494 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtgnp\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-kube-api-access-qtgnp\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938494 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938494 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938772 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938772 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938772 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-web-config\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.938772 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.938593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:16.979565 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:16.979525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qz7x7" event={"ID":"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9","Type":"ContainerStarted","Data":"5acaa573a00ea9f5b7df91c674a7f5de45f3f2b5a9ffd0c56c524ceb7009f0a0"}
Jul 24 12:54:17.039440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-out\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039811 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtgnp\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-kube-api-access-qtgnp\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039811 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039811 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039811 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039811 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.039811 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.039570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-web-config\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.042728 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.042699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.043046 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.043020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.043121 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.043027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.044264 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.044163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.044765 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.044476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.044765 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.044557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-web-config\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.044765 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.044703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.044984 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.044846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.045352 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.045316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-out\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.046011 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.045984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.046692 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.046670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.047115 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.047095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.056040 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.055967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtgnp\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-kube-api-access-qtgnp\") pod \"alertmanager-main-0\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.125903 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.125869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:54:17.281773 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.281739    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:54:17.284749 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:54:17.284709    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7d1b66_b21b_46cb_a2c2_c25b559586ec.slice/crio-476e7238e2e568663bd59eeb3c49c54bffc3c13076995b7fe33a03af24c06769 WatchSource:0}: Error finding container 476e7238e2e568663bd59eeb3c49c54bffc3c13076995b7fe33a03af24c06769: Status 404 returned error can't find the container with id 476e7238e2e568663bd59eeb3c49c54bffc3c13076995b7fe33a03af24c06769
Jul 24 12:54:17.988166 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.987949    2579 generic.go:356] "Generic (PLEG): container finished" podID="7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9" containerID="28c4fd9eb6397938cae49b7f3341e987b8326146c368313873781a315c6fbf59" exitCode=0
Jul 24 12:54:17.988166 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.988093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qz7x7" event={"ID":"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9","Type":"ContainerDied","Data":"28c4fd9eb6397938cae49b7f3341e987b8326146c368313873781a315c6fbf59"}
Jul 24 12:54:17.990506 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:17.990435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerStarted","Data":"476e7238e2e568663bd59eeb3c49c54bffc3c13076995b7fe33a03af24c06769"}
Jul 24 12:54:21.959287 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:21.959250    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hp2hl"
Jul 24 12:54:22.373573 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:22.372563    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:22.373573 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:22.373334    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:22.378532 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:22.378503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:23.016059 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:23.016031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:54:26.027593 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:26.027506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qz7x7" event={"ID":"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9","Type":"ContainerStarted","Data":"e7a7a43dca5e7e56f47885d4d11fea8d8715227848e9c28fcf6a4e9f1ee8f440"}
Jul 24 12:54:26.031732 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:26.031029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-wrsk5" event={"ID":"ab5c680f-75a8-4c64-a9cc-70ce761974dd","Type":"ContainerStarted","Data":"79f35967a50306cfd1eba22b373926a5499156e35a0e9bef82f0520ffe0df37f"}
Jul 24 12:54:26.031732 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:26.031661    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-wrsk5"
Jul 24 12:54:26.033030 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:26.032999    2579 patch_prober.go:28] interesting pod/downloads-7c97dc4555-wrsk5 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.13:8080/\": dial tcp 10.133.0.13:8080: connect: connection refused" start-of-body=
Jul 24 12:54:26.033146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:26.033059    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-wrsk5" podUID="ab5c680f-75a8-4c64-a9cc-70ce761974dd" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.13:8080/\": dial tcp 10.133.0.13:8080: connect: connection refused"
Jul 24 12:54:26.034584 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:26.034555    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerID="791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f" exitCode=0
Jul 24 12:54:26.034672 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:26.034616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerDied","Data":"791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f"}
Jul 24 12:54:26.059599 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:26.059539    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-wrsk5" podStartSLOduration=5.11932085 podStartE2EDuration="23.059519763s" totalImagesPullingTime="17.940198913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:26.057424764 +0000 UTC m=+90.046743459" watchObservedRunningTime="2026-07-24 12:54:26.059519763 +0000 UTC m=+90.048838460"
Jul 24 12:54:27.045309 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:27.045116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qz7x7" event={"ID":"7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9","Type":"ContainerStarted","Data":"2c271493b916610fa08c08cfe660ce741778c23216a42c9820a289cb248fa951"}
Jul 24 12:54:27.059975 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:27.059918    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-wrsk5"
Jul 24 12:54:27.068982 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:27.068166    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qz7x7" podStartSLOduration=11.034736457 podStartE2EDuration="12.068145532s" totalImagesPullingTime="951.868245ms" totalInitContainerRuntime="81.54083ms" isStatefulPod=true podCreationTimestamp="2026-07-24 12:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:27.066815055 +0000 UTC m=+91.056133750" watchObservedRunningTime="2026-07-24 12:54:27.068145532 +0000 UTC m=+91.057464232"
Jul 24 12:54:27.905642 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:27.905568    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:54:30.060776 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:30.060737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz" event={"ID":"45e0ad25-3162-4774-9887-15487b5e03bc","Type":"ContainerStarted","Data":"458acec6030d308051ed4be22fad61f37302077219666dd47b0161cfead5a726"}
Jul 24 12:54:30.064649 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:30.064616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerStarted","Data":"a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600"}
Jul 24 12:54:30.064792 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:30.064693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerStarted","Data":"7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772"}
Jul 24 12:54:30.064792 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:30.064732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerStarted","Data":"6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09"}
Jul 24 12:54:30.064792 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:30.064758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerStarted","Data":"3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b"}
Jul 24 12:54:30.064792 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:30.064783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerStarted","Data":"115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea"}
Jul 24 12:54:30.081601 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:30.081492    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c69fd7f4-kfgnz" podStartSLOduration=6.767330648 podStartE2EDuration="28.081471893s" totalImagesPullingTime="21.314141245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:30.079265473 +0000 UTC m=+94.068584169" watchObservedRunningTime="2026-07-24 12:54:30.081471893 +0000 UTC m=+94.070790589"
Jul 24 12:54:30.949728 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:30.949698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9d76c648c-5m9xs"
Jul 24 12:54:32.078338 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:32.078297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerStarted","Data":"52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c"}
Jul 24 12:54:32.116094 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:32.116028    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.122403972 podStartE2EDuration="16.116009994s" totalImagesPullingTime="12.960801846s" totalInitContainerRuntime="32.804176ms" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:32.113895007 +0000 UTC m=+96.103213705" watchObservedRunningTime="2026-07-24 12:54:32.116009994 +0000 UTC m=+96.105328699"
Jul 24 12:54:32.919178 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:32.919131    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b" podUID="3fbe6497-426b-41d3-a613-a792e28d5d68" containerName="registry" containerID="cri-o://bc0dffacbec78a34e606a61af60023b16edf3bd6bbf1c5370926919cfdcc136e" gracePeriod=30
Jul 24 12:54:33.085162 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.085131    2579 generic.go:356] "Generic (PLEG): container finished" podID="3fbe6497-426b-41d3-a613-a792e28d5d68" containerID="bc0dffacbec78a34e606a61af60023b16edf3bd6bbf1c5370926919cfdcc136e" exitCode=0
Jul 24 12:54:33.085786 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.085757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b" event={"ID":"3fbe6497-426b-41d3-a613-a792e28d5d68","Type":"ContainerDied","Data":"bc0dffacbec78a34e606a61af60023b16edf3bd6bbf1c5370926919cfdcc136e"}
Jul 24 12:54:33.230084 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.230052    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:54:33.302186 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.302153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-image-registry-private-configuration\") pod \"3fbe6497-426b-41d3-a613-a792e28d5d68\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") "
Jul 24 12:54:33.302371 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.302202    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") pod \"3fbe6497-426b-41d3-a613-a792e28d5d68\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") "
Jul 24 12:54:33.302371 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.302234    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f8p4\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-kube-api-access-2f8p4\") pod \"3fbe6497-426b-41d3-a613-a792e28d5d68\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") "
Jul 24 12:54:33.302371 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.302257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-trusted-ca\") pod \"3fbe6497-426b-41d3-a613-a792e28d5d68\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") "
Jul 24 12:54:33.302371 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.302280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-installation-pull-secrets\") pod \"3fbe6497-426b-41d3-a613-a792e28d5d68\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") "
Jul 24 12:54:33.302371 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.302312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fbe6497-426b-41d3-a613-a792e28d5d68-ca-trust-extracted\") pod \"3fbe6497-426b-41d3-a613-a792e28d5d68\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") "
Jul 24 12:54:33.303277 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.302974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3fbe6497-426b-41d3-a613-a792e28d5d68" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:54:33.303277 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.303051    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-certificates\") pod \"3fbe6497-426b-41d3-a613-a792e28d5d68\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") "
Jul 24 12:54:33.303277 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.303110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-bound-sa-token\") pod \"3fbe6497-426b-41d3-a613-a792e28d5d68\" (UID: \"3fbe6497-426b-41d3-a613-a792e28d5d68\") "
Jul 24 12:54:33.303277 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.303243    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3fbe6497-426b-41d3-a613-a792e28d5d68" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:54:33.303575 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.303372    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-trusted-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:54:33.303575 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.303392    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-certificates\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:54:33.305723 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.305691    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3fbe6497-426b-41d3-a613-a792e28d5d68" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:54:33.307024 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.306202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3fbe6497-426b-41d3-a613-a792e28d5d68" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:54:33.307024 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.306265    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3fbe6497-426b-41d3-a613-a792e28d5d68" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:54:33.307405 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.307376    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3fbe6497-426b-41d3-a613-a792e28d5d68" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:54:33.307650 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.307626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-kube-api-access-2f8p4" (OuterVolumeSpecName: "kube-api-access-2f8p4") pod "3fbe6497-426b-41d3-a613-a792e28d5d68" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68"). InnerVolumeSpecName "kube-api-access-2f8p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:54:33.312491 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.312451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbe6497-426b-41d3-a613-a792e28d5d68-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3fbe6497-426b-41d3-a613-a792e28d5d68" (UID: "3fbe6497-426b-41d3-a613-a792e28d5d68"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:54:33.403915 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.403873    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-image-registry-private-configuration\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:54:33.403915 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.403915    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-registry-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:54:33.404164 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.403949    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f8p4\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-kube-api-access-2f8p4\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:54:33.404164 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.403962    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fbe6497-426b-41d3-a613-a792e28d5d68-installation-pull-secrets\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:54:33.404164 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.403976    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fbe6497-426b-41d3-a613-a792e28d5d68-ca-trust-extracted\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:54:33.404164 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:33.403990    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fbe6497-426b-41d3-a613-a792e28d5d68-bound-sa-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:54:34.093385 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:34.093360    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b"
Jul 24 12:54:34.093801 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:34.093360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-745ff49dc7-zm67b" event={"ID":"3fbe6497-426b-41d3-a613-a792e28d5d68","Type":"ContainerDied","Data":"cddff5b6aa67fa5ce166fb4c032ae51485e7bfd5c4603fd0739151531eb227d3"}
Jul 24 12:54:34.093865 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:34.093830    2579 scope.go:117] "RemoveContainer" containerID="bc0dffacbec78a34e606a61af60023b16edf3bd6bbf1c5370926919cfdcc136e"
Jul 24 12:54:34.118493 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:34.118470    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-745ff49dc7-zm67b"]
Jul 24 12:54:34.130700 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:34.130669    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-745ff49dc7-zm67b"]
Jul 24 12:54:34.563651 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:34.563612    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbe6497-426b-41d3-a613-a792e28d5d68" path="/var/lib/kubelet/pods/3fbe6497-426b-41d3-a613-a792e28d5d68/volumes"
Jul 24 12:54:37.339640 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:37.339605    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77697885cd-25q5t"]
Jul 24 12:54:39.937042 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:54:39.937009    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nn8v9"
Jul 24 12:55:02.358537 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.358474    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77697885cd-25q5t" podUID="b4f5e4eb-0640-4125-998b-4fb450044413" containerName="console" containerID="cri-o://ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe" gracePeriod=15
Jul 24 12:55:02.639509 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.639484    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:55:02.749797 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.749765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c5p9\" (UniqueName: \"kubernetes.io/projected/b4f5e4eb-0640-4125-998b-4fb450044413-kube-api-access-6c5p9\") pod \"b4f5e4eb-0640-4125-998b-4fb450044413\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") "
Jul 24 12:55:02.750002 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.749826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-oauth-serving-cert\") pod \"b4f5e4eb-0640-4125-998b-4fb450044413\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") "
Jul 24 12:55:02.750002 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.749852    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-console-config\") pod \"b4f5e4eb-0640-4125-998b-4fb450044413\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") "
Jul 24 12:55:02.750002 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.749879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-trusted-ca-bundle\") pod \"b4f5e4eb-0640-4125-998b-4fb450044413\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") "
Jul 24 12:55:02.750002 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.749917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-oauth-config\") pod \"b4f5e4eb-0640-4125-998b-4fb450044413\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") "
Jul 24 12:55:02.750002 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.749961    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-service-ca\") pod \"b4f5e4eb-0640-4125-998b-4fb450044413\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") "
Jul 24 12:55:02.750250 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.750074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-serving-cert\") pod \"b4f5e4eb-0640-4125-998b-4fb450044413\" (UID: \"b4f5e4eb-0640-4125-998b-4fb450044413\") "
Jul 24 12:55:02.750392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.750371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4f5e4eb-0640-4125-998b-4fb450044413" (UID: "b4f5e4eb-0640-4125-998b-4fb450044413"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:02.750448 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.750383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4f5e4eb-0640-4125-998b-4fb450044413" (UID: "b4f5e4eb-0640-4125-998b-4fb450044413"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:02.750448 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.750393    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-console-config" (OuterVolumeSpecName: "console-config") pod "b4f5e4eb-0640-4125-998b-4fb450044413" (UID: "b4f5e4eb-0640-4125-998b-4fb450044413"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:02.750448 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.750372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b4f5e4eb-0640-4125-998b-4fb450044413" (UID: "b4f5e4eb-0640-4125-998b-4fb450044413"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:02.752281 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.752255    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4f5e4eb-0640-4125-998b-4fb450044413" (UID: "b4f5e4eb-0640-4125-998b-4fb450044413"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:02.752281 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.752263    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f5e4eb-0640-4125-998b-4fb450044413-kube-api-access-6c5p9" (OuterVolumeSpecName: "kube-api-access-6c5p9") pod "b4f5e4eb-0640-4125-998b-4fb450044413" (UID: "b4f5e4eb-0640-4125-998b-4fb450044413"). InnerVolumeSpecName "kube-api-access-6c5p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:02.752281 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.752275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4f5e4eb-0640-4125-998b-4fb450044413" (UID: "b4f5e4eb-0640-4125-998b-4fb450044413"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:02.851607 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.851574    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-oauth-serving-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:02.851607 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.851601    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-console-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:02.851607 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.851611    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-trusted-ca-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:02.851808 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.851618    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-oauth-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:02.851808 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.851627    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f5e4eb-0640-4125-998b-4fb450044413-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:02.851808 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.851634    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4f5e4eb-0640-4125-998b-4fb450044413-console-serving-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:02.851808 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:02.851641    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c5p9\" (UniqueName: \"kubernetes.io/projected/b4f5e4eb-0640-4125-998b-4fb450044413-kube-api-access-6c5p9\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:03.198327 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.198291    2579 generic.go:356] "Generic (PLEG): container finished" podID="b4f5e4eb-0640-4125-998b-4fb450044413" containerID="ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe" exitCode=0
Jul 24 12:55:03.198490 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.198343    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77697885cd-25q5t"
Jul 24 12:55:03.198490 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.198380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77697885cd-25q5t" event={"ID":"b4f5e4eb-0640-4125-998b-4fb450044413","Type":"ContainerDied","Data":"ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe"}
Jul 24 12:55:03.198490 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.198427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77697885cd-25q5t" event={"ID":"b4f5e4eb-0640-4125-998b-4fb450044413","Type":"ContainerDied","Data":"d6fa94903c0600bcdd124d3c6c85888df6c5b2b799055396609e22cb9cc22490"}
Jul 24 12:55:03.198490 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.198447    2579 scope.go:117] "RemoveContainer" containerID="ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe"
Jul 24 12:55:03.207158 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.207138    2579 scope.go:117] "RemoveContainer" containerID="ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe"
Jul 24 12:55:03.207441 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:55:03.207423    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe\": container with ID starting with ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe not found: ID does not exist" containerID="ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe"
Jul 24 12:55:03.207488 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.207450    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe"} err="failed to get container status \"ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe\": rpc error: code = NotFound desc = could not find container \"ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe\": container with ID starting with ee33cb32edae173ab2113c65b7a55db524ab87f26fbf907c9bb08980395147fe not found: ID does not exist"
Jul 24 12:55:03.220289 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.220258    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77697885cd-25q5t"]
Jul 24 12:55:03.226170 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:03.226149    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77697885cd-25q5t"]
Jul 24 12:55:04.562147 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:04.562111    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f5e4eb-0640-4125-998b-4fb450044413" path="/var/lib/kubelet/pods/b4f5e4eb-0640-4125-998b-4fb450044413/volumes"
Jul 24 12:55:36.193709 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:36.193674    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:55:36.315571 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:36.315498    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="alertmanager" containerID="cri-o://115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea" gracePeriod=120
Jul 24 12:55:36.315771 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:36.315566    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy-metric" containerID="cri-o://a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600" gracePeriod=120
Jul 24 12:55:36.315771 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:36.315600    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy-web" containerID="cri-o://6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09" gracePeriod=120
Jul 24 12:55:36.315771 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:36.315622    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="config-reloader" containerID="cri-o://3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b" gracePeriod=120
Jul 24 12:55:36.315771 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:36.315611    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy" containerID="cri-o://7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772" gracePeriod=120
Jul 24 12:55:36.315771 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:36.315609    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="prom-label-proxy" containerID="cri-o://52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c" gracePeriod=120
Jul 24 12:55:37.322042 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322010    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerID="52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c" exitCode=0
Jul 24 12:55:37.322042 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322037    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerID="a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600" exitCode=0
Jul 24 12:55:37.322042 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322047    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerID="7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772" exitCode=0
Jul 24 12:55:37.322466 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322057    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerID="3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b" exitCode=0
Jul 24 12:55:37.322466 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322065    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerID="115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea" exitCode=0
Jul 24 12:55:37.322466 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerDied","Data":"52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c"}
Jul 24 12:55:37.322466 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerDied","Data":"a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600"}
Jul 24 12:55:37.322466 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerDied","Data":"7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772"}
Jul 24 12:55:37.322466 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerDied","Data":"3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b"}
Jul 24 12:55:37.322466 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.322218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerDied","Data":"115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea"}
Jul 24 12:55:37.556870 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.556844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:37.601966 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.601855    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-tls-assets\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.601966 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.601896    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtgnp\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-kube-api-access-qtgnp\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.601966 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.601947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-metrics-client-ca\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.601974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-out\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.601998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-main-tls\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-main-db\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602085    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-web-config\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602126    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-trusted-ca-bundle\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602156    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602180    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-volume\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602202    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-cluster-tls-config\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.602243 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy\") pod \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\" (UID: \"1b7d1b66-b21b-46cb-a2c2-c25b559586ec\") "
Jul 24 12:55:37.603059 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.602743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:37.603176 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.603099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:37.606844 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.606816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:37.607078 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.607057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:37.608275 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.608238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:37.608496 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.608467    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-out" (OuterVolumeSpecName: "config-out") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:37.608693 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.608662    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-kube-api-access-qtgnp" (OuterVolumeSpecName: "kube-api-access-qtgnp") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "kube-api-access-qtgnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:37.608947 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.608898    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:37.609214 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.609190    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:37.609569 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.609518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-volume" (OuterVolumeSpecName: "config-volume") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:37.610215 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.610191    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:37.611742 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.611642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:37.622328 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.622305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-web-config" (OuterVolumeSpecName: "web-config") pod "1b7d1b66-b21b-46cb-a2c2-c25b559586ec" (UID: "1b7d1b66-b21b-46cb-a2c2-c25b559586ec"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:37.702881 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702854    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-main-db\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.702881 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702881    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702892    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-web-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702901    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702911    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702938    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-volume\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702946    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-cluster-tls-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702953    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702961    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-tls-assets\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702969    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtgnp\" (UniqueName: \"kubernetes.io/projected/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-kube-api-access-qtgnp\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702976    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-metrics-client-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702984    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-config-out\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.703061 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:37.702991    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b7d1b66-b21b-46cb-a2c2-c25b559586ec-secret-alertmanager-main-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:55:38.163069 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163029    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-765676c455-srm8t"]
Jul 24 12:55:38.163348 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163331    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="prom-label-proxy"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163350    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="prom-label-proxy"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163360    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="init-config-reloader"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="init-config-reloader"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163377    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy-web"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163384    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy-web"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163397    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163406    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163419    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="config-reloader"
Jul 24 12:55:38.163435 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163427    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="config-reloader"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163439    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbe6497-426b-41d3-a613-a792e28d5d68" containerName="registry"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163447    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbe6497-426b-41d3-a613-a792e28d5d68" containerName="registry"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163459    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy-metric"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163468    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy-metric"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163480    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="alertmanager"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163488    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="alertmanager"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163503    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f5e4eb-0640-4125-998b-4fb450044413" containerName="console"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163511    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f5e4eb-0640-4125-998b-4fb450044413" containerName="console"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163572    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f5e4eb-0640-4125-998b-4fb450044413" containerName="console"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163583    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy-metric"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163592    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="alertmanager"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163601    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy-web"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163610    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="kube-rbac-proxy"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163618    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="prom-label-proxy"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163626    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbe6497-426b-41d3-a613-a792e28d5d68" containerName="registry"
Jul 24 12:55:38.163832 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.163634    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerName="config-reloader"
Jul 24 12:55:38.166740 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.166720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.171562 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.169323    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 12:55:38.172334 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.172308    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 12:55:38.172452 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.172333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 12:55:38.172452 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.172333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 12:55:38.172452 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.172396    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 12:55:38.172572 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.172495    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b8dbp\""
Jul 24 12:55:38.177962 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.177825    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 12:55:38.180491 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.180473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765676c455-srm8t"]
Jul 24 12:55:38.206986 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.206957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-trusted-ca-bundle\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.206986 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.206990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-serving-cert\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.207168 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.207017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-oauth-serving-cert\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.207168 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.207089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sk44\" (UniqueName: \"kubernetes.io/projected/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-kube-api-access-2sk44\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.207168 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.207142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-oauth-config\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.207275 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.207182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-service-ca\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.207275 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.207199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-config\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.307882 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.307847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sk44\" (UniqueName: \"kubernetes.io/projected/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-kube-api-access-2sk44\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.307882 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.307888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-oauth-config\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.308148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.307907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-service-ca\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.308148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.308047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-config\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.308148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.308098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-trusted-ca-bundle\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.308148 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.308140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-serving-cert\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.308311 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.308192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-oauth-serving-cert\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.308665 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.308638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-config\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.308799 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.308705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-service-ca\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.308799 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.308772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-oauth-serving-cert\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.309086 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.309066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-trusted-ca-bundle\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.311001 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.310982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-oauth-config\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.311085 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.311072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-serving-cert\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.317027 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.317011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sk44\" (UniqueName: \"kubernetes.io/projected/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-kube-api-access-2sk44\") pod \"console-765676c455-srm8t\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") " pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.329103 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.329077    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" containerID="6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09" exitCode=0
Jul 24 12:55:38.329392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.329152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerDied","Data":"6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09"}
Jul 24 12:55:38.329392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.329182    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.329392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.329221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b7d1b66-b21b-46cb-a2c2-c25b559586ec","Type":"ContainerDied","Data":"476e7238e2e568663bd59eeb3c49c54bffc3c13076995b7fe33a03af24c06769"}
Jul 24 12:55:38.329392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.329253    2579 scope.go:117] "RemoveContainer" containerID="52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c"
Jul 24 12:55:38.336619 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.336599    2579 scope.go:117] "RemoveContainer" containerID="a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600"
Jul 24 12:55:38.344151 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.344135    2579 scope.go:117] "RemoveContainer" containerID="7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772"
Jul 24 12:55:38.350498 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.350482    2579 scope.go:117] "RemoveContainer" containerID="6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09"
Jul 24 12:55:38.357143 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.356752    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:55:38.357588 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.357557    2579 scope.go:117] "RemoveContainer" containerID="3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b"
Jul 24 12:55:38.364637 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.364620    2579 scope.go:117] "RemoveContainer" containerID="115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea"
Jul 24 12:55:38.365002 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.364916    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:55:38.371484 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.371467    2579 scope.go:117] "RemoveContainer" containerID="791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f"
Jul 24 12:55:38.378128 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.378112    2579 scope.go:117] "RemoveContainer" containerID="52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c"
Jul 24 12:55:38.378350 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:55:38.378328    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c\": container with ID starting with 52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c not found: ID does not exist" containerID="52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c"
Jul 24 12:55:38.378395 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.378365    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c"} err="failed to get container status \"52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c\": rpc error: code = NotFound desc = could not find container \"52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c\": container with ID starting with 52458855f6a6b18a1cdf2802f91fb3a41d6ee7c58a0f84df400a5b18684fb93c not found: ID does not exist"
Jul 24 12:55:38.378395 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.378383    2579 scope.go:117] "RemoveContainer" containerID="a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600"
Jul 24 12:55:38.378611 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:55:38.378594    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600\": container with ID starting with a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600 not found: ID does not exist" containerID="a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600"
Jul 24 12:55:38.378656 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.378615    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600"} err="failed to get container status \"a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600\": rpc error: code = NotFound desc = could not find container \"a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600\": container with ID starting with a82e936b54500b726c30f931f6f6198bb1b346b23556d70a17a02ac544d4c600 not found: ID does not exist"
Jul 24 12:55:38.378656 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.378629    2579 scope.go:117] "RemoveContainer" containerID="7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772"
Jul 24 12:55:38.378824 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:55:38.378809    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772\": container with ID starting with 7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772 not found: ID does not exist" containerID="7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772"
Jul 24 12:55:38.378865 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.378827    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772"} err="failed to get container status \"7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772\": rpc error: code = NotFound desc = could not find container \"7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772\": container with ID starting with 7f5210afa95b06b2c56e926bb55b008a956d89d4700b92acf6c1bfe859e25772 not found: ID does not exist"
Jul 24 12:55:38.378865 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.378839    2579 scope.go:117] "RemoveContainer" containerID="6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09"
Jul 24 12:55:38.379079 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:55:38.379064    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09\": container with ID starting with 6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09 not found: ID does not exist" containerID="6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09"
Jul 24 12:55:38.379121 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.379085    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09"} err="failed to get container status \"6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09\": rpc error: code = NotFound desc = could not find container \"6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09\": container with ID starting with 6997df1ba2279812108cc6d3dda65edf9b61cf0e464fc7c00a7e53c8ea319b09 not found: ID does not exist"
Jul 24 12:55:38.379121 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.379100    2579 scope.go:117] "RemoveContainer" containerID="3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b"
Jul 24 12:55:38.379320 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:55:38.379304    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b\": container with ID starting with 3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b not found: ID does not exist" containerID="3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b"
Jul 24 12:55:38.379363 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.379322    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b"} err="failed to get container status \"3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b\": rpc error: code = NotFound desc = could not find container \"3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b\": container with ID starting with 3e351242672cdf36968dffb1a49f0157e036e42b0221ab638a4e922ba2fe4b5b not found: ID does not exist"
Jul 24 12:55:38.379363 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.379333    2579 scope.go:117] "RemoveContainer" containerID="115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea"
Jul 24 12:55:38.379566 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:55:38.379550    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea\": container with ID starting with 115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea not found: ID does not exist" containerID="115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea"
Jul 24 12:55:38.379606 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.379570    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea"} err="failed to get container status \"115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea\": rpc error: code = NotFound desc = could not find container \"115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea\": container with ID starting with 115576cd338d9b56a0faca364a10a1591d4f2969c45f9e0e59d9a5907e4209ea not found: ID does not exist"
Jul 24 12:55:38.379606 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.379585    2579 scope.go:117] "RemoveContainer" containerID="791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f"
Jul 24 12:55:38.379789 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:55:38.379775    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f\": container with ID starting with 791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f not found: ID does not exist" containerID="791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f"
Jul 24 12:55:38.379826 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.379793    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f"} err="failed to get container status \"791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f\": rpc error: code = NotFound desc = could not find container \"791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f\": container with ID starting with 791550bfb1bc398dde37ee86002b5f7d27ec0376a9b6480e97c11bff2b4ac45f not found: ID does not exist"
Jul 24 12:55:38.406593 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.406568    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:55:38.410591 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.410574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.413446 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.413398    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 12:55:38.413533 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.413484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 12:55:38.413744 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.413728    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 12:55:38.414127 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.414111    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 12:55:38.414224 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.414204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 12:55:38.414500 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.414486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 12:55:38.414541 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.414514    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 12:55:38.415133 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.415117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 12:55:38.415216 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.415149    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jp87l\""
Jul 24 12:55:38.422674 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.422659    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 12:55:38.430026 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.430002    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:55:38.481556 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.481523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:38.509879 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.509853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-web-config\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.509879 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.509887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.509940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/537d218b-c2da-43bf-bb18-e641c0d7e26c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.509962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcl9\" (UniqueName: \"kubernetes.io/projected/537d218b-c2da-43bf-bb18-e641c0d7e26c-kube-api-access-6tcl9\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.509978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-config-volume\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.509991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/537d218b-c2da-43bf-bb18-e641c0d7e26c-config-out\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.510010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.510040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510102 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.510089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/537d218b-c2da-43bf-bb18-e641c0d7e26c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.510112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.510133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.510227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/537d218b-c2da-43bf-bb18-e641c0d7e26c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.510392 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.510291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/537d218b-c2da-43bf-bb18-e641c0d7e26c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.563062 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.563031    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7d1b66-b21b-46cb-a2c2-c25b559586ec" path="/var/lib/kubelet/pods/1b7d1b66-b21b-46cb-a2c2-c25b559586ec/volumes"
Jul 24 12:55:38.608898 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.608829    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765676c455-srm8t"]
Jul 24 12:55:38.610847 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.610821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/537d218b-c2da-43bf-bb18-e641c0d7e26c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.610938 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.610858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-web-config\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.610938 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.610884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.610938 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.610917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/537d218b-c2da-43bf-bb18-e641c0d7e26c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611099 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.610956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcl9\" (UniqueName: \"kubernetes.io/projected/537d218b-c2da-43bf-bb18-e641c0d7e26c-kube-api-access-6tcl9\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611099 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.610981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-config-volume\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611099 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.611002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/537d218b-c2da-43bf-bb18-e641c0d7e26c-config-out\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611099 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.611027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611099 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.611049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611099 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.611093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/537d218b-c2da-43bf-bb18-e641c0d7e26c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611347 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:55:38.611104    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c9b00f_a03d_42e5_a53d_fea52b8dfb08.slice/crio-2d4aa1b3eec0e0411f62fa903336aaaa68d94202c95dc53a2b105c6ff11cb71f WatchSource:0}: Error finding container 2d4aa1b3eec0e0411f62fa903336aaaa68d94202c95dc53a2b105c6ff11cb71f: Status 404 returned error can't find the container with id 2d4aa1b3eec0e0411f62fa903336aaaa68d94202c95dc53a2b105c6ff11cb71f
Jul 24 12:55:38.611347 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.611132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611452 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.611428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.611524 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.611509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/537d218b-c2da-43bf-bb18-e641c0d7e26c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.612504 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.611976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/537d218b-c2da-43bf-bb18-e641c0d7e26c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.613560 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.612710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/537d218b-c2da-43bf-bb18-e641c0d7e26c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.613560 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.613318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/537d218b-c2da-43bf-bb18-e641c0d7e26c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.613903 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.613878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/537d218b-c2da-43bf-bb18-e641c0d7e26c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.614873 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.614851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-config-volume\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.614996 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.614954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.615327 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.615304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.615566 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.615537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-web-config\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.615685 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.615669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/537d218b-c2da-43bf-bb18-e641c0d7e26c-config-out\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.616505 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.616484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.616682 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.616658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.617316 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.617294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/537d218b-c2da-43bf-bb18-e641c0d7e26c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.621213 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.621188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcl9\" (UniqueName: \"kubernetes.io/projected/537d218b-c2da-43bf-bb18-e641c0d7e26c-kube-api-access-6tcl9\") pod \"alertmanager-main-0\" (UID: \"537d218b-c2da-43bf-bb18-e641c0d7e26c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.719023 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.718991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:55:38.862197 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:38.862166    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:55:38.865281 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:55:38.865247    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537d218b_c2da_43bf_bb18_e641c0d7e26c.slice/crio-08979a4fedcdf31185db83b1eb8b4d44bfb6cf14f4ed4817cc283911093acffb WatchSource:0}: Error finding container 08979a4fedcdf31185db83b1eb8b4d44bfb6cf14f4ed4817cc283911093acffb: Status 404 returned error can't find the container with id 08979a4fedcdf31185db83b1eb8b4d44bfb6cf14f4ed4817cc283911093acffb
Jul 24 12:55:39.334726 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:39.334638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765676c455-srm8t" event={"ID":"40c9b00f-a03d-42e5-a53d-fea52b8dfb08","Type":"ContainerStarted","Data":"b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e"}
Jul 24 12:55:39.334726 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:39.334691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765676c455-srm8t" event={"ID":"40c9b00f-a03d-42e5-a53d-fea52b8dfb08","Type":"ContainerStarted","Data":"2d4aa1b3eec0e0411f62fa903336aaaa68d94202c95dc53a2b105c6ff11cb71f"}
Jul 24 12:55:39.336813 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:39.336786    2579 generic.go:356] "Generic (PLEG): container finished" podID="537d218b-c2da-43bf-bb18-e641c0d7e26c" containerID="2bcc2fc2d3b56c345c9593654de004fa2b6f70236fff0b40e5755bf4003dbd9f" exitCode=0
Jul 24 12:55:39.336939 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:39.336830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"537d218b-c2da-43bf-bb18-e641c0d7e26c","Type":"ContainerDied","Data":"2bcc2fc2d3b56c345c9593654de004fa2b6f70236fff0b40e5755bf4003dbd9f"}
Jul 24 12:55:39.336939 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:39.336887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"537d218b-c2da-43bf-bb18-e641c0d7e26c","Type":"ContainerStarted","Data":"08979a4fedcdf31185db83b1eb8b4d44bfb6cf14f4ed4817cc283911093acffb"}
Jul 24 12:55:39.366083 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:39.366040    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-765676c455-srm8t" podStartSLOduration=1.366024187 podStartE2EDuration="1.366024187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:55:39.36461253 +0000 UTC m=+163.353931227" watchObservedRunningTime="2026-07-24 12:55:39.366024187 +0000 UTC m=+163.355342884"
Jul 24 12:55:40.345711 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:40.345672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"537d218b-c2da-43bf-bb18-e641c0d7e26c","Type":"ContainerStarted","Data":"b5e22e0272616091c4c4d01fbcac50aeffb56596f686c960daa2d0bb4db07acb"}
Jul 24 12:55:40.346224 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:40.345785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"537d218b-c2da-43bf-bb18-e641c0d7e26c","Type":"ContainerStarted","Data":"28f692f18c1bafa00f6023890ad0bce59ef4866f6c0fbd5d3875835dda24e53a"}
Jul 24 12:55:40.346224 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:40.345821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"537d218b-c2da-43bf-bb18-e641c0d7e26c","Type":"ContainerStarted","Data":"4fc450c8b42367735515c8bf58cff44fdf3ec40361dbc8d7a026185090f4290d"}
Jul 24 12:55:40.346224 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:40.345857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"537d218b-c2da-43bf-bb18-e641c0d7e26c","Type":"ContainerStarted","Data":"5725f6f0daeb23366c7cb73ba3abf8217bd91b59766e1783b25c814f08e5095a"}
Jul 24 12:55:40.346224 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:40.345893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"537d218b-c2da-43bf-bb18-e641c0d7e26c","Type":"ContainerStarted","Data":"d3fb89fbf69f797f7e70a04fa1c3eadfd5275159e27d5a2a547928d94698db29"}
Jul 24 12:55:40.346224 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:40.345939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"537d218b-c2da-43bf-bb18-e641c0d7e26c","Type":"ContainerStarted","Data":"62ae4d53cc1d4b71e7142b2163cd68c6b5dad45deef455964e252b4d1a464f58"}
Jul 24 12:55:40.384673 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:40.384618    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.370693412 podStartE2EDuration="2.384595833s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.902421ms" isStatefulPod=false podCreationTimestamp="2026-07-24 12:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:55:40.381611905 +0000 UTC m=+164.370930611" watchObservedRunningTime="2026-07-24 12:55:40.384595833 +0000 UTC m=+164.373914529"
Jul 24 12:55:48.481987 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:48.481875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:48.482406 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:48.481999    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:48.486744 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:48.486721    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:55:49.384884 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:55:49.384840    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765676c455-srm8t"
Jul 24 12:57:07.281241 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.281194    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"]
Jul 24 12:57:07.289201 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.289174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.291976 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.291941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 12:57:07.292126 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.292023    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wbkjc\""
Jul 24 12:57:07.292126 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.292047    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 12:57:07.297318 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.297295    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"]
Jul 24 12:57:07.400744 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.400704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.400744 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.400745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltq7\" (UniqueName: \"kubernetes.io/projected/5eab5181-fcbd-4230-8849-e0ea680f1de8-kube-api-access-qltq7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.400992 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.400766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.502179 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.502145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.502179 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.502183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltq7\" (UniqueName: \"kubernetes.io/projected/5eab5181-fcbd-4230-8849-e0ea680f1de8-kube-api-access-qltq7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.502396 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.502204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.502523 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.502507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.502602 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.502583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.513270 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.513229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltq7\" (UniqueName: \"kubernetes.io/projected/5eab5181-fcbd-4230-8849-e0ea680f1de8-kube-api-access-qltq7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.598228 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.598140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:07.731764 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:07.731733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"]
Jul 24 12:57:07.734836 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:57:07.734808    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eab5181_fcbd_4230_8849_e0ea680f1de8.slice/crio-35791896903b2b5696a58d4444ee74dbcd2c952e6db00eea4f0d981017a54e8a WatchSource:0}: Error finding container 35791896903b2b5696a58d4444ee74dbcd2c952e6db00eea4f0d981017a54e8a: Status 404 returned error can't find the container with id 35791896903b2b5696a58d4444ee74dbcd2c952e6db00eea4f0d981017a54e8a
Jul 24 12:57:08.662567 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:08.662530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v" event={"ID":"5eab5181-fcbd-4230-8849-e0ea680f1de8","Type":"ContainerStarted","Data":"35791896903b2b5696a58d4444ee74dbcd2c952e6db00eea4f0d981017a54e8a"}
Jul 24 12:57:13.686910 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:13.686873    2579 generic.go:356] "Generic (PLEG): container finished" podID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerID="c2e78e70a8afb98edd3b83f743d25a7cb0d27295a6841d1df1031f3914aab524" exitCode=0
Jul 24 12:57:13.686910 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:13.686912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v" event={"ID":"5eab5181-fcbd-4230-8849-e0ea680f1de8","Type":"ContainerDied","Data":"c2e78e70a8afb98edd3b83f743d25a7cb0d27295a6841d1df1031f3914aab524"}
Jul 24 12:57:15.697515 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:15.697465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v" event={"ID":"5eab5181-fcbd-4230-8849-e0ea680f1de8","Type":"ContainerStarted","Data":"61416ca15e60eea81403e184e416664d7dcbf72aa9312a243d8adb76ded5a68a"}
Jul 24 12:57:16.702844 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:16.702809    2579 generic.go:356] "Generic (PLEG): container finished" podID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerID="61416ca15e60eea81403e184e416664d7dcbf72aa9312a243d8adb76ded5a68a" exitCode=0
Jul 24 12:57:16.703345 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:16.702867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v" event={"ID":"5eab5181-fcbd-4230-8849-e0ea680f1de8","Type":"ContainerDied","Data":"61416ca15e60eea81403e184e416664d7dcbf72aa9312a243d8adb76ded5a68a"}
Jul 24 12:57:23.732021 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:23.731988    2579 generic.go:356] "Generic (PLEG): container finished" podID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerID="e09c0296506509db101924d74bbbdceecf53ffc453831beb5b4298a97e1fae98" exitCode=0
Jul 24 12:57:23.732420 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:23.732064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v" event={"ID":"5eab5181-fcbd-4230-8849-e0ea680f1de8","Type":"ContainerDied","Data":"e09c0296506509db101924d74bbbdceecf53ffc453831beb5b4298a97e1fae98"}
Jul 24 12:57:24.860794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:24.860770    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:24.955005 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:24.954966    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-util\") pod \"5eab5181-fcbd-4230-8849-e0ea680f1de8\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") "
Jul 24 12:57:24.955204 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:24.955070    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qltq7\" (UniqueName: \"kubernetes.io/projected/5eab5181-fcbd-4230-8849-e0ea680f1de8-kube-api-access-qltq7\") pod \"5eab5181-fcbd-4230-8849-e0ea680f1de8\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") "
Jul 24 12:57:24.955204 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:24.955114    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-bundle\") pod \"5eab5181-fcbd-4230-8849-e0ea680f1de8\" (UID: \"5eab5181-fcbd-4230-8849-e0ea680f1de8\") "
Jul 24 12:57:24.955888 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:24.955847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-bundle" (OuterVolumeSpecName: "bundle") pod "5eab5181-fcbd-4230-8849-e0ea680f1de8" (UID: "5eab5181-fcbd-4230-8849-e0ea680f1de8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:57:24.957383 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:24.957359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eab5181-fcbd-4230-8849-e0ea680f1de8-kube-api-access-qltq7" (OuterVolumeSpecName: "kube-api-access-qltq7") pod "5eab5181-fcbd-4230-8849-e0ea680f1de8" (UID: "5eab5181-fcbd-4230-8849-e0ea680f1de8"). InnerVolumeSpecName "kube-api-access-qltq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:57:24.959356 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:24.959332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-util" (OuterVolumeSpecName: "util") pod "5eab5181-fcbd-4230-8849-e0ea680f1de8" (UID: "5eab5181-fcbd-4230-8849-e0ea680f1de8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:57:25.056281 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:25.056195    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:25.056281 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:25.056223    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eab5181-fcbd-4230-8849-e0ea680f1de8-util\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:25.056281 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:25.056232    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltq7\" (UniqueName: \"kubernetes.io/projected/5eab5181-fcbd-4230-8849-e0ea680f1de8-kube-api-access-qltq7\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:25.742326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:25.742293    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v"
Jul 24 12:57:25.742326 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:25.742318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pn92v" event={"ID":"5eab5181-fcbd-4230-8849-e0ea680f1de8","Type":"ContainerDied","Data":"35791896903b2b5696a58d4444ee74dbcd2c952e6db00eea4f0d981017a54e8a"}
Jul 24 12:57:25.742584 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:25.742363    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35791896903b2b5696a58d4444ee74dbcd2c952e6db00eea4f0d981017a54e8a"
Jul 24 12:57:34.783503 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.783461    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"]
Jul 24 12:57:34.784604 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.784581    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerName="util"
Jul 24 12:57:34.784604 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.784605    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerName="util"
Jul 24 12:57:34.784715 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.784614    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerName="extract"
Jul 24 12:57:34.784715 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.784620    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerName="extract"
Jul 24 12:57:34.784715 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.784629    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerName="pull"
Jul 24 12:57:34.784715 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.784634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerName="pull"
Jul 24 12:57:34.784715 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.784689    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eab5181-fcbd-4230-8849-e0ea680f1de8" containerName="extract"
Jul 24 12:57:34.818117 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.818073    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8sw8k"]
Jul 24 12:57:34.818319 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.818228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.822991 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.822968    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 12:57:34.823797 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.823776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wbkjc\""
Jul 24 12:57:34.824174 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.824147    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 12:57:34.831314 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.831286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlc9\" (UniqueName: \"kubernetes.io/projected/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-kube-api-access-qnlc9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.831445 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.831328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.831445 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.831372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.831445 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.831418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"]
Jul 24 12:57:34.831551 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.831452    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8sw8k"]
Jul 24 12:57:34.831551 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.831523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:34.835226 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.835190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 12:57:34.835356 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.835265    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 12:57:34.848374 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.848345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-v8tk4\""
Jul 24 12:57:34.932123 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.932083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlc9\" (UniqueName: \"kubernetes.io/projected/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-kube-api-access-qnlc9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.932123 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.932130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v8vz\" (UniqueName: \"kubernetes.io/projected/0e634168-9b83-4724-8991-b15cb11aa574-kube-api-access-8v8vz\") pod \"cert-manager-webhook-6486ffd997-8sw8k\" (UID: \"0e634168-9b83-4724-8991-b15cb11aa574\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:34.932396 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.932155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.932396 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.932208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.932396 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.932239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e634168-9b83-4724-8991-b15cb11aa574-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8sw8k\" (UID: \"0e634168-9b83-4724-8991-b15cb11aa574\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:34.932567 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.932550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.932615 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.932548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:34.945403 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:34.945371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlc9\" (UniqueName: \"kubernetes.io/projected/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-kube-api-access-qnlc9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:35.032699 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.032660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v8vz\" (UniqueName: \"kubernetes.io/projected/0e634168-9b83-4724-8991-b15cb11aa574-kube-api-access-8v8vz\") pod \"cert-manager-webhook-6486ffd997-8sw8k\" (UID: \"0e634168-9b83-4724-8991-b15cb11aa574\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:35.032879 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.032732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e634168-9b83-4724-8991-b15cb11aa574-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8sw8k\" (UID: \"0e634168-9b83-4724-8991-b15cb11aa574\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:35.049509 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.049438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v8vz\" (UniqueName: \"kubernetes.io/projected/0e634168-9b83-4724-8991-b15cb11aa574-kube-api-access-8v8vz\") pod \"cert-manager-webhook-6486ffd997-8sw8k\" (UID: \"0e634168-9b83-4724-8991-b15cb11aa574\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:35.052547 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.052519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e634168-9b83-4724-8991-b15cb11aa574-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8sw8k\" (UID: \"0e634168-9b83-4724-8991-b15cb11aa574\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:35.127092 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.127050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:35.139431 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.139402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:35.297388 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.297358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"]
Jul 24 12:57:35.299145 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:57:35.299116    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc2fd26_62fc_4e4e_8a66_5a7d8657680e.slice/crio-b04745bb963f87b03d5e5c06eacd7c9fb8f2cff2931a1f49e91a4680441aaa33 WatchSource:0}: Error finding container b04745bb963f87b03d5e5c06eacd7c9fb8f2cff2931a1f49e91a4680441aaa33: Status 404 returned error can't find the container with id b04745bb963f87b03d5e5c06eacd7c9fb8f2cff2931a1f49e91a4680441aaa33
Jul 24 12:57:35.320523 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.320492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8sw8k"]
Jul 24 12:57:35.323347 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:57:35.323313    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e634168_9b83_4724_8991_b15cb11aa574.slice/crio-68654f0d90b1073bdec40ac400432465ed9da645e3036fb2c0d7f1f9ac8b0ca3 WatchSource:0}: Error finding container 68654f0d90b1073bdec40ac400432465ed9da645e3036fb2c0d7f1f9ac8b0ca3: Status 404 returned error can't find the container with id 68654f0d90b1073bdec40ac400432465ed9da645e3036fb2c0d7f1f9ac8b0ca3
Jul 24 12:57:35.778403 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.778369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k" event={"ID":"0e634168-9b83-4724-8991-b15cb11aa574","Type":"ContainerStarted","Data":"68654f0d90b1073bdec40ac400432465ed9da645e3036fb2c0d7f1f9ac8b0ca3"}
Jul 24 12:57:35.779778 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.779747    2579 generic.go:356] "Generic (PLEG): container finished" podID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerID="8d643065ee2ebb73b184e64e259125e6db2c857ac974364006fd40c859a53152" exitCode=0
Jul 24 12:57:35.779778 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.779784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d" event={"ID":"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e","Type":"ContainerDied","Data":"8d643065ee2ebb73b184e64e259125e6db2c857ac974364006fd40c859a53152"}
Jul 24 12:57:35.779966 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:35.779816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d" event={"ID":"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e","Type":"ContainerStarted","Data":"b04745bb963f87b03d5e5c06eacd7c9fb8f2cff2931a1f49e91a4680441aaa33"}
Jul 24 12:57:39.798457 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:39.798419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k" event={"ID":"0e634168-9b83-4724-8991-b15cb11aa574","Type":"ContainerStarted","Data":"5d1dfa1ef7ae9ac6c53e9512dcc27edd5d71448da7294c4d70eba538481c7c94"}
Jul 24 12:57:39.798945 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:39.798499    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:39.800185 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:39.800157    2579 generic.go:356] "Generic (PLEG): container finished" podID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerID="38e3bed65f718d48f3d065acf72b98fc982c67894ddfeda4dad7ddfc65d68838" exitCode=0
Jul 24 12:57:39.800275 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:39.800193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d" event={"ID":"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e","Type":"ContainerDied","Data":"38e3bed65f718d48f3d065acf72b98fc982c67894ddfeda4dad7ddfc65d68838"}
Jul 24 12:57:39.818215 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:39.818164    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k" podStartSLOduration=2.372753246 podStartE2EDuration="5.818148588s" totalImagesPullingTime="3.445395342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:39.817184828 +0000 UTC m=+283.806503522" watchObservedRunningTime="2026-07-24 12:57:39.818148588 +0000 UTC m=+283.807467284"
Jul 24 12:57:40.806029 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:40.805990    2579 generic.go:356] "Generic (PLEG): container finished" podID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerID="d7c685163cdebd95bc41198925bbb93ea66d9122bd7f6708c5f3a7e3c96f0e83" exitCode=0
Jul 24 12:57:40.806426 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:40.806091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d" event={"ID":"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e","Type":"ContainerDied","Data":"d7c685163cdebd95bc41198925bbb93ea66d9122bd7f6708c5f3a7e3c96f0e83"}
Jul 24 12:57:41.937861 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:41.937837    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:41.978724 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:41.978691    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-bundle\") pod \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") "
Jul 24 12:57:41.978915 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:41.978742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-util\") pod \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") "
Jul 24 12:57:41.978915 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:41.978775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnlc9\" (UniqueName: \"kubernetes.io/projected/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-kube-api-access-qnlc9\") pod \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\" (UID: \"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e\") "
Jul 24 12:57:41.979188 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:41.979160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-bundle" (OuterVolumeSpecName: "bundle") pod "fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" (UID: "fbc2fd26-62fc-4e4e-8a66-5a7d8657680e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:57:41.981076 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:41.981053    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-kube-api-access-qnlc9" (OuterVolumeSpecName: "kube-api-access-qnlc9") pod "fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" (UID: "fbc2fd26-62fc-4e4e-8a66-5a7d8657680e"). InnerVolumeSpecName "kube-api-access-qnlc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:57:41.983080 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:41.983054    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-util" (OuterVolumeSpecName: "util") pod "fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" (UID: "fbc2fd26-62fc-4e4e-8a66-5a7d8657680e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:57:42.079569 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:42.079475    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-util\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:42.079569 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:42.079510    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnlc9\" (UniqueName: \"kubernetes.io/projected/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-kube-api-access-qnlc9\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:42.079569 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:42.079522    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc2fd26-62fc-4e4e-8a66-5a7d8657680e-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:42.815819 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:42.815790    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d"
Jul 24 12:57:42.816014 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:42.815778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc684d" event={"ID":"fbc2fd26-62fc-4e4e-8a66-5a7d8657680e","Type":"ContainerDied","Data":"b04745bb963f87b03d5e5c06eacd7c9fb8f2cff2931a1f49e91a4680441aaa33"}
Jul 24 12:57:42.816014 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:42.815913    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04745bb963f87b03d5e5c06eacd7c9fb8f2cff2931a1f49e91a4680441aaa33"
Jul 24 12:57:45.808445 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:45.808416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8sw8k"
Jul 24 12:57:47.804617 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.804584    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"]
Jul 24 12:57:47.805007 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.804852    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerName="util"
Jul 24 12:57:47.805007 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.804863    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerName="util"
Jul 24 12:57:47.805007 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.804875    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerName="pull"
Jul 24 12:57:47.805007 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.804882    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerName="pull"
Jul 24 12:57:47.805007 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.804890    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerName="extract"
Jul 24 12:57:47.805007 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.804895    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerName="extract"
Jul 24 12:57:47.805007 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.804974    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc2fd26-62fc-4e4e-8a66-5a7d8657680e" containerName="extract"
Jul 24 12:57:47.806693 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.806677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"
Jul 24 12:57:47.811085 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.811056    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:57:47.811189 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.811143    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-x7c9h\""
Jul 24 12:57:47.812131 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.812106    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 12:57:47.815262 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.815238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8fb660-7f39-4724-b1d4-b89e57bcf4c7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gd5tn\" (UID: \"6c8fb660-7f39-4724-b1d4-b89e57bcf4c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"
Jul 24 12:57:47.815369 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.815294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54jbq\" (UniqueName: \"kubernetes.io/projected/6c8fb660-7f39-4724-b1d4-b89e57bcf4c7-kube-api-access-54jbq\") pod \"openshift-lws-operator-fd8ccff4c-gd5tn\" (UID: \"6c8fb660-7f39-4724-b1d4-b89e57bcf4c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"
Jul 24 12:57:47.820354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.820330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"]
Jul 24 12:57:47.916794 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.916751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8fb660-7f39-4724-b1d4-b89e57bcf4c7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gd5tn\" (UID: \"6c8fb660-7f39-4724-b1d4-b89e57bcf4c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"
Jul 24 12:57:47.917122 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.917102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54jbq\" (UniqueName: \"kubernetes.io/projected/6c8fb660-7f39-4724-b1d4-b89e57bcf4c7-kube-api-access-54jbq\") pod \"openshift-lws-operator-fd8ccff4c-gd5tn\" (UID: \"6c8fb660-7f39-4724-b1d4-b89e57bcf4c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"
Jul 24 12:57:47.917972 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.917946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8fb660-7f39-4724-b1d4-b89e57bcf4c7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gd5tn\" (UID: \"6c8fb660-7f39-4724-b1d4-b89e57bcf4c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"
Jul 24 12:57:47.927467 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:47.927443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54jbq\" (UniqueName: \"kubernetes.io/projected/6c8fb660-7f39-4724-b1d4-b89e57bcf4c7-kube-api-access-54jbq\") pod \"openshift-lws-operator-fd8ccff4c-gd5tn\" (UID: \"6c8fb660-7f39-4724-b1d4-b89e57bcf4c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"
Jul 24 12:57:48.116354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:48.116263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"
Jul 24 12:57:48.249120 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:48.249095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn"]
Jul 24 12:57:48.251745 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:57:48.251714    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c8fb660_7f39_4724_b1d4_b89e57bcf4c7.slice/crio-582eed6ba933363dd5ef10733743e25295b3a6677e1a016c8a5833c09c44a50e WatchSource:0}: Error finding container 582eed6ba933363dd5ef10733743e25295b3a6677e1a016c8a5833c09c44a50e: Status 404 returned error can't find the container with id 582eed6ba933363dd5ef10733743e25295b3a6677e1a016c8a5833c09c44a50e
Jul 24 12:57:48.841864 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:48.841810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn" event={"ID":"6c8fb660-7f39-4724-b1d4-b89e57bcf4c7","Type":"ContainerStarted","Data":"582eed6ba933363dd5ef10733743e25295b3a6677e1a016c8a5833c09c44a50e"}
Jul 24 12:57:50.851311 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:50.851274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn" event={"ID":"6c8fb660-7f39-4724-b1d4-b89e57bcf4c7","Type":"ContainerStarted","Data":"faba0f833827f0fcef88e80efe22e8bdc5c2aca2598c7b41474fd24983d837e8"}
Jul 24 12:57:50.871718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:50.871666    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd5tn" podStartSLOduration=2.301158337 podStartE2EDuration="3.871649924s" totalImagesPullingTime="1.570491587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:50.870293251 +0000 UTC m=+294.859611947" watchObservedRunningTime="2026-07-24 12:57:50.871649924 +0000 UTC m=+294.860968621"
Jul 24 12:57:53.482839 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.482801    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"]
Jul 24 12:57:53.485275 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.485254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.487865 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.487845    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wbkjc\""
Jul 24 12:57:53.487979 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.487878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 12:57:53.488801 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.488783    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 12:57:53.499407 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.499383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"]
Jul 24 12:57:53.567501 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.567452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpg7\" (UniqueName: \"kubernetes.io/projected/124a11a0-f301-4a21-b95b-7f253ccc87f3-kube-api-access-9zpg7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.567696 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.567508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.567696 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.567572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.668219 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.668183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.668219 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.668224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.668411 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.668288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpg7\" (UniqueName: \"kubernetes.io/projected/124a11a0-f301-4a21-b95b-7f253ccc87f3-kube-api-access-9zpg7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.668598 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.668577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.668660 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.668607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.678668 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.678634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpg7\" (UniqueName: \"kubernetes.io/projected/124a11a0-f301-4a21-b95b-7f253ccc87f3-kube-api-access-9zpg7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.793798 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.793716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:53.926024 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:53.925988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"]
Jul 24 12:57:53.928734 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:57:53.928704    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124a11a0_f301_4a21_b95b_7f253ccc87f3.slice/crio-d707759f5aa5963365d1a2c198a0d5ac26f5ec242dce380756b60cc73b2fea38 WatchSource:0}: Error finding container d707759f5aa5963365d1a2c198a0d5ac26f5ec242dce380756b60cc73b2fea38: Status 404 returned error can't find the container with id d707759f5aa5963365d1a2c198a0d5ac26f5ec242dce380756b60cc73b2fea38
Jul 24 12:57:54.868496 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:54.868460    2579 generic.go:356] "Generic (PLEG): container finished" podID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerID="eeaa062e0db00a05b6d4334c055202282e476f4483af47251d5c11880a51ac59" exitCode=0
Jul 24 12:57:54.868900 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:54.868548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4" event={"ID":"124a11a0-f301-4a21-b95b-7f253ccc87f3","Type":"ContainerDied","Data":"eeaa062e0db00a05b6d4334c055202282e476f4483af47251d5c11880a51ac59"}
Jul 24 12:57:54.868900 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:54.868605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4" event={"ID":"124a11a0-f301-4a21-b95b-7f253ccc87f3","Type":"ContainerStarted","Data":"d707759f5aa5963365d1a2c198a0d5ac26f5ec242dce380756b60cc73b2fea38"}
Jul 24 12:57:55.874601 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:55.874505    2579 generic.go:356] "Generic (PLEG): container finished" podID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerID="8437dfc32702877668cabe9428a682adad56182d090a0a6eec294d3b18d1b134" exitCode=0
Jul 24 12:57:55.874601 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:55.874577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4" event={"ID":"124a11a0-f301-4a21-b95b-7f253ccc87f3","Type":"ContainerDied","Data":"8437dfc32702877668cabe9428a682adad56182d090a0a6eec294d3b18d1b134"}
Jul 24 12:57:56.394112 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:56.394088    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 12:57:56.880497 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:56.880464    2579 generic.go:356] "Generic (PLEG): container finished" podID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerID="7028284ae02344013d27be89c85ae2efbc401879622f4bacdd6f174e92a41e97" exitCode=0
Jul 24 12:57:56.881739 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:56.880546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4" event={"ID":"124a11a0-f301-4a21-b95b-7f253ccc87f3","Type":"ContainerDied","Data":"7028284ae02344013d27be89c85ae2efbc401879622f4bacdd6f174e92a41e97"}
Jul 24 12:57:58.006671 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.006645    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:58.103121 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.103087    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zpg7\" (UniqueName: \"kubernetes.io/projected/124a11a0-f301-4a21-b95b-7f253ccc87f3-kube-api-access-9zpg7\") pod \"124a11a0-f301-4a21-b95b-7f253ccc87f3\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") "
Jul 24 12:57:58.103333 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.103185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-bundle\") pod \"124a11a0-f301-4a21-b95b-7f253ccc87f3\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") "
Jul 24 12:57:58.103333 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.103228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-util\") pod \"124a11a0-f301-4a21-b95b-7f253ccc87f3\" (UID: \"124a11a0-f301-4a21-b95b-7f253ccc87f3\") "
Jul 24 12:57:58.103895 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.103864    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-bundle" (OuterVolumeSpecName: "bundle") pod "124a11a0-f301-4a21-b95b-7f253ccc87f3" (UID: "124a11a0-f301-4a21-b95b-7f253ccc87f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:57:58.105389 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.105358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124a11a0-f301-4a21-b95b-7f253ccc87f3-kube-api-access-9zpg7" (OuterVolumeSpecName: "kube-api-access-9zpg7") pod "124a11a0-f301-4a21-b95b-7f253ccc87f3" (UID: "124a11a0-f301-4a21-b95b-7f253ccc87f3"). InnerVolumeSpecName "kube-api-access-9zpg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:57:58.109495 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.109401    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-util" (OuterVolumeSpecName: "util") pod "124a11a0-f301-4a21-b95b-7f253ccc87f3" (UID: "124a11a0-f301-4a21-b95b-7f253ccc87f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:57:58.204682 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.204644    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:58.204682 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.204673    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a11a0-f301-4a21-b95b-7f253ccc87f3-util\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:58.204682 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.204682    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zpg7\" (UniqueName: \"kubernetes.io/projected/124a11a0-f301-4a21-b95b-7f253ccc87f3-kube-api-access-9zpg7\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:57:58.889899 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.889806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4" event={"ID":"124a11a0-f301-4a21-b95b-7f253ccc87f3","Type":"ContainerDied","Data":"d707759f5aa5963365d1a2c198a0d5ac26f5ec242dce380756b60cc73b2fea38"}
Jul 24 12:57:58.889899 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.889825    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkbzg4"
Jul 24 12:57:58.889899 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:57:58.889859    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d707759f5aa5963365d1a2c198a0d5ac26f5ec242dce380756b60cc73b2fea38"
Jul 24 12:58:10.792250 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.792214    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"]
Jul 24 12:58:10.792718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.792490    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerName="extract"
Jul 24 12:58:10.792718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.792500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerName="extract"
Jul 24 12:58:10.792718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.792510    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerName="pull"
Jul 24 12:58:10.792718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.792515    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerName="pull"
Jul 24 12:58:10.792718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.792532    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerName="util"
Jul 24 12:58:10.792718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.792537    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerName="util"
Jul 24 12:58:10.792718 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.792584    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="124a11a0-f301-4a21-b95b-7f253ccc87f3" containerName="extract"
Jul 24 12:58:10.795084 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.795069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:10.797771 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.797748    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 12:58:10.800793 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.800771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pmmpb\""
Jul 24 12:58:10.801661 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.801644    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 12:58:10.802122 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.802105    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 12:58:10.803506 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.803490    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 12:58:10.816266 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.816245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"]
Jul 24 12:58:10.906939 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.906903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1292528-bf0c-44b9-84d7-a8c29bc244fe-manifests\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:10.907110 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.906969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1292528-bf0c-44b9-84d7-a8c29bc244fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:10.907110 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.907011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1292528-bf0c-44b9-84d7-a8c29bc244fe-tmp\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:10.907110 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.907033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1292528-bf0c-44b9-84d7-a8c29bc244fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:10.907212 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:10.907106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs728\" (UniqueName: \"kubernetes.io/projected/a1292528-bf0c-44b9-84d7-a8c29bc244fe-kube-api-access-bs728\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.007660 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.007616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs728\" (UniqueName: \"kubernetes.io/projected/a1292528-bf0c-44b9-84d7-a8c29bc244fe-kube-api-access-bs728\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.007861 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.007716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1292528-bf0c-44b9-84d7-a8c29bc244fe-manifests\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.007861 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.007743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1292528-bf0c-44b9-84d7-a8c29bc244fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.007861 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.007767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1292528-bf0c-44b9-84d7-a8c29bc244fe-tmp\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.007861 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.007792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1292528-bf0c-44b9-84d7-a8c29bc244fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.008210 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.008187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1292528-bf0c-44b9-84d7-a8c29bc244fe-manifests\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.010184 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.010157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1292528-bf0c-44b9-84d7-a8c29bc244fe-tmp\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.010385 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.010366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1292528-bf0c-44b9-84d7-a8c29bc244fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.010447 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.010419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1292528-bf0c-44b9-84d7-a8c29bc244fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.017848 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.017826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs728\" (UniqueName: \"kubernetes.io/projected/a1292528-bf0c-44b9-84d7-a8c29bc244fe-kube-api-access-bs728\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-99vcd\" (UID: \"a1292528-bf0c-44b9-84d7-a8c29bc244fe\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.104946 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.104838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:11.243600 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.243571    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"]
Jul 24 12:58:11.245774 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:58:11.245737    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1292528_bf0c_44b9_84d7_a8c29bc244fe.slice/crio-7296783aefe9bd8efadd4bcfbbd3c314c88dc181a263290d28d510b607151737 WatchSource:0}: Error finding container 7296783aefe9bd8efadd4bcfbbd3c314c88dc181a263290d28d510b607151737: Status 404 returned error can't find the container with id 7296783aefe9bd8efadd4bcfbbd3c314c88dc181a263290d28d510b607151737
Jul 24 12:58:11.247620 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.247603    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:58:11.942294 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:11.942253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd" event={"ID":"a1292528-bf0c-44b9-84d7-a8c29bc244fe","Type":"ContainerStarted","Data":"7296783aefe9bd8efadd4bcfbbd3c314c88dc181a263290d28d510b607151737"}
Jul 24 12:58:14.957988 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:14.957949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd" event={"ID":"a1292528-bf0c-44b9-84d7-a8c29bc244fe","Type":"ContainerStarted","Data":"d8bc1b2f584f1d4beb2795f63b187bbdde15e2e78506fa3779a99906436ffe54"}
Jul 24 12:58:15.963597 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:15.963565    2579 generic.go:356] "Generic (PLEG): container finished" podID="a1292528-bf0c-44b9-84d7-a8c29bc244fe" containerID="d8bc1b2f584f1d4beb2795f63b187bbdde15e2e78506fa3779a99906436ffe54" exitCode=0
Jul 24 12:58:15.964083 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:15.963618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd" event={"ID":"a1292528-bf0c-44b9-84d7-a8c29bc244fe","Type":"ContainerDied","Data":"d8bc1b2f584f1d4beb2795f63b187bbdde15e2e78506fa3779a99906436ffe54"}
Jul 24 12:58:16.975019 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:16.974905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd" event={"ID":"a1292528-bf0c-44b9-84d7-a8c29bc244fe","Type":"ContainerStarted","Data":"f73b5ef1f0426def8f284843397e7276f78133fd02ab0770ba57401627d9e230"}
Jul 24 12:58:16.975376 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:16.975047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:17.000659 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:17.000611    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd" podStartSLOduration=2.185575056 podStartE2EDuration="7.000595853s" totalImagesPullingTime="3.798315238s" totalInitContainerRuntime="1.016705559s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:58:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:58:16.998261616 +0000 UTC m=+320.987580313" watchObservedRunningTime="2026-07-24 12:58:17.000595853 +0000 UTC m=+320.989914550"
Jul 24 12:58:27.981749 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:27.981718    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-99vcd"
Jul 24 12:58:31.127305 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.127269    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gj9rt"]
Jul 24 12:58:31.132642 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.132622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:31.135485 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.135463    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 24 12:58:31.136330 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.136311    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7hrqm\""
Jul 24 12:58:31.143579 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.143559    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gj9rt"]
Jul 24 12:58:31.277473 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.277441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kp9\" (UniqueName: \"kubernetes.io/projected/57112a2b-02a9-43e2-9fa1-e717c92a4798-kube-api-access-64kp9\") pod \"odh-model-controller-7998c67dc4-gj9rt\" (UID: \"57112a2b-02a9-43e2-9fa1-e717c92a4798\") " pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:31.277473 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.277480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57112a2b-02a9-43e2-9fa1-e717c92a4798-cert\") pod \"odh-model-controller-7998c67dc4-gj9rt\" (UID: \"57112a2b-02a9-43e2-9fa1-e717c92a4798\") " pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:31.378327 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.378244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57112a2b-02a9-43e2-9fa1-e717c92a4798-cert\") pod \"odh-model-controller-7998c67dc4-gj9rt\" (UID: \"57112a2b-02a9-43e2-9fa1-e717c92a4798\") " pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:31.378327 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.378321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kp9\" (UniqueName: \"kubernetes.io/projected/57112a2b-02a9-43e2-9fa1-e717c92a4798-kube-api-access-64kp9\") pod \"odh-model-controller-7998c67dc4-gj9rt\" (UID: \"57112a2b-02a9-43e2-9fa1-e717c92a4798\") " pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:31.378509 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:58:31.378396    2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 24 12:58:31.378509 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:58:31.378470    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57112a2b-02a9-43e2-9fa1-e717c92a4798-cert podName:57112a2b-02a9-43e2-9fa1-e717c92a4798 nodeName:}" failed. No retries permitted until 2026-07-24 12:58:31.878452442 +0000 UTC m=+335.867771116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/57112a2b-02a9-43e2-9fa1-e717c92a4798-cert") pod "odh-model-controller-7998c67dc4-gj9rt" (UID: "57112a2b-02a9-43e2-9fa1-e717c92a4798") : secret "odh-model-controller-webhook-cert" not found
Jul 24 12:58:31.390214 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.390191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kp9\" (UniqueName: \"kubernetes.io/projected/57112a2b-02a9-43e2-9fa1-e717c92a4798-kube-api-access-64kp9\") pod \"odh-model-controller-7998c67dc4-gj9rt\" (UID: \"57112a2b-02a9-43e2-9fa1-e717c92a4798\") " pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:31.882103 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.882067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57112a2b-02a9-43e2-9fa1-e717c92a4798-cert\") pod \"odh-model-controller-7998c67dc4-gj9rt\" (UID: \"57112a2b-02a9-43e2-9fa1-e717c92a4798\") " pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:31.884617 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:31.884598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57112a2b-02a9-43e2-9fa1-e717c92a4798-cert\") pod \"odh-model-controller-7998c67dc4-gj9rt\" (UID: \"57112a2b-02a9-43e2-9fa1-e717c92a4798\") " pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:32.042484 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:32.042450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:32.178866 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:32.178836    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gj9rt"]
Jul 24 12:58:32.183244 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:58:32.183217    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57112a2b_02a9_43e2_9fa1_e717c92a4798.slice/crio-8361eca0b250346ba23a04440cee8fe2812ae70d27abe58b6eb73f231d24eef8 WatchSource:0}: Error finding container 8361eca0b250346ba23a04440cee8fe2812ae70d27abe58b6eb73f231d24eef8: Status 404 returned error can't find the container with id 8361eca0b250346ba23a04440cee8fe2812ae70d27abe58b6eb73f231d24eef8
Jul 24 12:58:33.048142 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:33.048075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt" event={"ID":"57112a2b-02a9-43e2-9fa1-e717c92a4798","Type":"ContainerStarted","Data":"8361eca0b250346ba23a04440cee8fe2812ae70d27abe58b6eb73f231d24eef8"}
Jul 24 12:58:35.059815 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:35.059786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt" event={"ID":"57112a2b-02a9-43e2-9fa1-e717c92a4798","Type":"ContainerStarted","Data":"1432f9af201b23096e27def3ca07892614f473b39f929a3d354edd0030fcc9ee"}
Jul 24 12:58:35.060213 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:35.059912    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:35.093553 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:35.093505    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt" podStartSLOduration=1.377705249 podStartE2EDuration="4.093490572s" totalImagesPullingTime="2.715785323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:58:35.092672467 +0000 UTC m=+339.081991163" watchObservedRunningTime="2026-07-24 12:58:35.093490572 +0000 UTC m=+339.082809266"
Jul 24 12:58:38.822017 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.821982    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"]
Jul 24 12:58:38.824171 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.824154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:38.829786 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.829763    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 12:58:38.830994 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.830970    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 12:58:38.831114 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.831055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 12:58:38.831267 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.831247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2pm28\""
Jul 24 12:58:38.844509 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.844485    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"]
Jul 24 12:58:38.941650 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.941614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7w8\" (UniqueName: \"kubernetes.io/projected/e55e2688-3ee0-4915-9bcf-14b4e683248a-kube-api-access-rz7w8\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:38.941831 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.941656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e55e2688-3ee0-4915-9bcf-14b4e683248a-manager-config\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:38.941831 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.941723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e55e2688-3ee0-4915-9bcf-14b4e683248a-cert\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:38.941831 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:38.941753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55e2688-3ee0-4915-9bcf-14b4e683248a-metrics-cert\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.042451 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.042408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7w8\" (UniqueName: \"kubernetes.io/projected/e55e2688-3ee0-4915-9bcf-14b4e683248a-kube-api-access-rz7w8\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.042451 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.042454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e55e2688-3ee0-4915-9bcf-14b4e683248a-manager-config\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.042711 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.042489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e55e2688-3ee0-4915-9bcf-14b4e683248a-cert\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.042711 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.042509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55e2688-3ee0-4915-9bcf-14b4e683248a-metrics-cert\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.043167 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.043147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e55e2688-3ee0-4915-9bcf-14b4e683248a-manager-config\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.045272 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.045252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e55e2688-3ee0-4915-9bcf-14b4e683248a-cert\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.045366 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.045287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55e2688-3ee0-4915-9bcf-14b4e683248a-metrics-cert\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.066637 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.066604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7w8\" (UniqueName: \"kubernetes.io/projected/e55e2688-3ee0-4915-9bcf-14b4e683248a-kube-api-access-rz7w8\") pod \"lws-controller-manager-756cc79bf9-hpgrp\" (UID: \"e55e2688-3ee0-4915-9bcf-14b4e683248a\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.133494 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.133411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:39.270009 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:39.269982    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"]
Jul 24 12:58:39.271432 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:58:39.271403    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55e2688_3ee0_4915_9bcf_14b4e683248a.slice/crio-1966e5123d48d3b7f808276b8c81bf013a64f5e4554768bc606c55c104cdae9e WatchSource:0}: Error finding container 1966e5123d48d3b7f808276b8c81bf013a64f5e4554768bc606c55c104cdae9e: Status 404 returned error can't find the container with id 1966e5123d48d3b7f808276b8c81bf013a64f5e4554768bc606c55c104cdae9e
Jul 24 12:58:40.084046 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:40.084004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp" event={"ID":"e55e2688-3ee0-4915-9bcf-14b4e683248a","Type":"ContainerStarted","Data":"1966e5123d48d3b7f808276b8c81bf013a64f5e4554768bc606c55c104cdae9e"}
Jul 24 12:58:41.090389 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:41.090300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp" event={"ID":"e55e2688-3ee0-4915-9bcf-14b4e683248a","Type":"ContainerStarted","Data":"51ec24516efaa3da259f7af8e845fe612f5c3037d7a98ab9338875f49eb13ca3"}
Jul 24 12:58:41.090742 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:41.090421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:41.161506 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:41.161436    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp" podStartSLOduration=1.596618711 podStartE2EDuration="3.161416409s" totalImagesPullingTime="1.564797698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:58:41.15953126 +0000 UTC m=+345.148849954" watchObservedRunningTime="2026-07-24 12:58:41.161416409 +0000 UTC m=+345.150735105"
Jul 24 12:58:46.067331 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:46.067299    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-gj9rt"
Jul 24 12:58:50.780758 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.780669    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"]
Jul 24 12:58:50.784226 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.784206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.787065 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.787048    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hw2cd\""
Jul 24 12:58:50.791833 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.791811    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 12:58:50.791961 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.791847    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 12:58:50.796572 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.796556    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 12:58:50.810908 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.810873    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"]
Jul 24 12:58:50.846766 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.846739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.846878 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.846769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.846878 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.846797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.846878 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.846869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsfh9\" (UniqueName: \"kubernetes.io/projected/05c7f37a-2a45-4cfa-941b-dc0ea0721541-kube-api-access-wsfh9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.847004 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.846914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.847004 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.846962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.847004 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.846986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.847004 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.847001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.847157 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.847016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948270 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948399 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948399 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948399 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfh9\" (UniqueName: \"kubernetes.io/projected/05c7f37a-2a45-4cfa-941b-dc0ea0721541-kube-api-access-wsfh9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948539 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948539 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948539 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948539 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948714 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948714 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.948966 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.948944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.949076 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.949034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.949076 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.949047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.949375 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.949347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.950907 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.950884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.951141 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.951126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.968045 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.968023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c7f37a-2a45-4cfa-941b-dc0ea0721541-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:50.968261 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:50.968242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfh9\" (UniqueName: \"kubernetes.io/projected/05c7f37a-2a45-4cfa-941b-dc0ea0721541-kube-api-access-wsfh9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w\" (UID: \"05c7f37a-2a45-4cfa-941b-dc0ea0721541\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:51.094065 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:51.093985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:51.250408 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:51.250192    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"]
Jul 24 12:58:51.253041 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:58:51.253013    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c7f37a_2a45_4cfa_941b_dc0ea0721541.slice/crio-108fa1f33ef59d6a4a00a11e9d1f5a94f45def855e2285206b84429eb1130a8c WatchSource:0}: Error finding container 108fa1f33ef59d6a4a00a11e9d1f5a94f45def855e2285206b84429eb1130a8c: Status 404 returned error can't find the container with id 108fa1f33ef59d6a4a00a11e9d1f5a94f45def855e2285206b84429eb1130a8c
Jul 24 12:58:52.097107 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:52.097072    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-hpgrp"
Jul 24 12:58:52.147644 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:52.147602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w" event={"ID":"05c7f37a-2a45-4cfa-941b-dc0ea0721541","Type":"ContainerStarted","Data":"108fa1f33ef59d6a4a00a11e9d1f5a94f45def855e2285206b84429eb1130a8c"}
Jul 24 12:58:54.223768 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:54.223717    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 12:58:54.224084 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:54.223844    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 12:58:54.224084 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:54.223885    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 12:58:55.160939 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:55.160883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w" event={"ID":"05c7f37a-2a45-4cfa-941b-dc0ea0721541","Type":"ContainerStarted","Data":"6c7ed7843fb66a1eb7c80c2e4eea1b07b27f617f50163d04dee3fb54696da0d5"}
Jul 24 12:58:55.189156 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:55.189106    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w" podStartSLOduration=2.220693323 podStartE2EDuration="5.18908884s" totalImagesPullingTime="2.968395517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:58:55.188111597 +0000 UTC m=+359.177430294" watchObservedRunningTime="2026-07-24 12:58:55.18908884 +0000 UTC m=+359.178407537"
Jul 24 12:58:56.094662 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:56.094628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:56.099589 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:56.099564    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:56.165318 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:56.165285    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:58:56.166308 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:58:56.166287    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w"
Jul 24 12:59:29.682618 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.682584    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f5c8d5d86-khztv"]
Jul 24 12:59:29.690455 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.690434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.716993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.715594    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5c8d5d86-khztv"]
Jul 24 12:59:29.873984 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.873954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-serving-cert\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.873984 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.873991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-oauth-serving-cert\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.874210 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.874008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-trusted-ca-bundle\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.874210 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.874065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-config\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.874210 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.874127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-oauth-config\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.874210 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.874157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-service-ca\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.874210 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.874185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8q9g\" (UniqueName: \"kubernetes.io/projected/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-kube-api-access-s8q9g\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975009 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.974909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-serving-cert\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975009 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.974964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-oauth-serving-cert\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975009 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.974987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-trusted-ca-bundle\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975254 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.975027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-config\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975254 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.975069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-oauth-config\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975254 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.975229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-service-ca\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975410 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.975282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8q9g\" (UniqueName: \"kubernetes.io/projected/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-kube-api-access-s8q9g\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975840 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.975804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-config\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.975969 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.975842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-trusted-ca-bundle\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.976020 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.975971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-oauth-serving-cert\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.976020 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.975985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-service-ca\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.977574 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.977559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-oauth-config\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.977659 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.977644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-console-serving-cert\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.987039 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.987010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8q9g\" (UniqueName: \"kubernetes.io/projected/48b225b2-12e6-4e6d-a3fa-b79be6875bfc-kube-api-access-s8q9g\") pod \"console-7f5c8d5d86-khztv\" (UID: \"48b225b2-12e6-4e6d-a3fa-b79be6875bfc\") " pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:29.999777 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:29.999758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:30.132808 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:30.132786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5c8d5d86-khztv"]
Jul 24 12:59:30.133904 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:59:30.133878    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b225b2_12e6_4e6d_a3fa_b79be6875bfc.slice/crio-d7dc45e3080f4c442076c6ba3215053f3c5e2372456bb2e38c21dd2183ed2075 WatchSource:0}: Error finding container d7dc45e3080f4c442076c6ba3215053f3c5e2372456bb2e38c21dd2183ed2075: Status 404 returned error can't find the container with id d7dc45e3080f4c442076c6ba3215053f3c5e2372456bb2e38c21dd2183ed2075
Jul 24 12:59:30.317621 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:30.317530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5c8d5d86-khztv" event={"ID":"48b225b2-12e6-4e6d-a3fa-b79be6875bfc","Type":"ContainerStarted","Data":"16e3eabd1686be6542c4cfee51ef88439e8c8a73f7f32328a184edc081d1d950"}
Jul 24 12:59:30.317621 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:30.317591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5c8d5d86-khztv" event={"ID":"48b225b2-12e6-4e6d-a3fa-b79be6875bfc","Type":"ContainerStarted","Data":"d7dc45e3080f4c442076c6ba3215053f3c5e2372456bb2e38c21dd2183ed2075"}
Jul 24 12:59:31.823476 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:31.823424    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f5c8d5d86-khztv" podStartSLOduration=2.823405447 podStartE2EDuration="2.823405447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:59:30.354639018 +0000 UTC m=+394.343957714" watchObservedRunningTime="2026-07-24 12:59:31.823405447 +0000 UTC m=+395.812724148"
Jul 24 12:59:31.824370 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:31.824350    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mcqm5"]
Jul 24 12:59:31.827472 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:31.827456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5"
Jul 24 12:59:31.833093 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:31.833069    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 12:59:31.833795 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:31.833776    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 12:59:31.834065 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:31.834051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-q5n78\""
Jul 24 12:59:31.840608 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:31.840583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mcqm5"]
Jul 24 12:59:31.993729 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:31.993699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnsv\" (UniqueName: \"kubernetes.io/projected/ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9-kube-api-access-nbnsv\") pod \"kuadrant-operator-catalog-mcqm5\" (UID: \"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9\") " pod="kuadrant-system/kuadrant-operator-catalog-mcqm5"
Jul 24 12:59:32.095108 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.095019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnsv\" (UniqueName: \"kubernetes.io/projected/ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9-kube-api-access-nbnsv\") pod \"kuadrant-operator-catalog-mcqm5\" (UID: \"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9\") " pod="kuadrant-system/kuadrant-operator-catalog-mcqm5"
Jul 24 12:59:32.105566 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.105532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnsv\" (UniqueName: \"kubernetes.io/projected/ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9-kube-api-access-nbnsv\") pod \"kuadrant-operator-catalog-mcqm5\" (UID: \"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9\") " pod="kuadrant-system/kuadrant-operator-catalog-mcqm5"
Jul 24 12:59:32.137277 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.137250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5"
Jul 24 12:59:32.151913 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.151876    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mcqm5"]
Jul 24 12:59:32.273346 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.273278    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mcqm5"]
Jul 24 12:59:32.275784 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:59:32.275761    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae309bc1_0a8c_4cc7_b4bd_58c4fc59ece9.slice/crio-d84597611d0c30c81ce7d285cc800e92ae012ba52e7a0855f8d22dc214f40e06 WatchSource:0}: Error finding container d84597611d0c30c81ce7d285cc800e92ae012ba52e7a0855f8d22dc214f40e06: Status 404 returned error can't find the container with id d84597611d0c30c81ce7d285cc800e92ae012ba52e7a0855f8d22dc214f40e06
Jul 24 12:59:32.328060 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.328027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5" event={"ID":"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9","Type":"ContainerStarted","Data":"d84597611d0c30c81ce7d285cc800e92ae012ba52e7a0855f8d22dc214f40e06"}
Jul 24 12:59:32.362800 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.362731    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7fbm6"]
Jul 24 12:59:32.366686 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.366670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:32.376285 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.376259    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7fbm6"]
Jul 24 12:59:32.498202 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.498159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzd9b\" (UniqueName: \"kubernetes.io/projected/0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8-kube-api-access-vzd9b\") pod \"kuadrant-operator-catalog-7fbm6\" (UID: \"0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8\") " pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:32.598731 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.598699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzd9b\" (UniqueName: \"kubernetes.io/projected/0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8-kube-api-access-vzd9b\") pod \"kuadrant-operator-catalog-7fbm6\" (UID: \"0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8\") " pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:32.608874 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.608851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzd9b\" (UniqueName: \"kubernetes.io/projected/0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8-kube-api-access-vzd9b\") pod \"kuadrant-operator-catalog-7fbm6\" (UID: \"0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8\") " pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:32.676042 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.676006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:32.819492 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:32.819457    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7fbm6"]
Jul 24 12:59:32.822578 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:59:32.822548    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd7bca9_3e8e_4148_a1fe_4d1d1a5d07d8.slice/crio-ce907ef7ca8c078bb41fcafc0db1ca830914b2fbb709ee0316d2eb3bc1da9047 WatchSource:0}: Error finding container ce907ef7ca8c078bb41fcafc0db1ca830914b2fbb709ee0316d2eb3bc1da9047: Status 404 returned error can't find the container with id ce907ef7ca8c078bb41fcafc0db1ca830914b2fbb709ee0316d2eb3bc1da9047
Jul 24 12:59:33.334080 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:33.334046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6" event={"ID":"0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8","Type":"ContainerStarted","Data":"ce907ef7ca8c078bb41fcafc0db1ca830914b2fbb709ee0316d2eb3bc1da9047"}
Jul 24 12:59:35.345193 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.345096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5" event={"ID":"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9","Type":"ContainerStarted","Data":"c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45"}
Jul 24 12:59:35.345193 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.345163    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5" podUID="ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9" containerName="registry-server" containerID="cri-o://c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45" gracePeriod=2
Jul 24 12:59:35.346804 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.346778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6" event={"ID":"0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8","Type":"ContainerStarted","Data":"5b176a8c21ae5ce8c7390691bf9d273e787de450ba023299f3a61f306855c6d8"}
Jul 24 12:59:35.362941 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.362888    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5" podStartSLOduration=2.292560244 podStartE2EDuration="4.362877045s" totalImagesPullingTime="2.070316801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:59:35.362123376 +0000 UTC m=+399.351442072" watchObservedRunningTime="2026-07-24 12:59:35.362877045 +0000 UTC m=+399.352195740"
Jul 24 12:59:35.379914 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.379862    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6" podStartSLOduration=1.853655382 podStartE2EDuration="3.379845945s" totalImagesPullingTime="1.526190563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:59:35.379208385 +0000 UTC m=+399.368527103" watchObservedRunningTime="2026-07-24 12:59:35.379845945 +0000 UTC m=+399.369164643"
Jul 24 12:59:35.595401 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.595342    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5"
Jul 24 12:59:35.727230 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.727197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbnsv\" (UniqueName: \"kubernetes.io/projected/ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9-kube-api-access-nbnsv\") pod \"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9\" (UID: \"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9\") "
Jul 24 12:59:35.729440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.729413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9-kube-api-access-nbnsv" (OuterVolumeSpecName: "kube-api-access-nbnsv") pod "ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9" (UID: "ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9"). InnerVolumeSpecName "kube-api-access-nbnsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:59:35.828542 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:35.828510    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnsv\" (UniqueName: \"kubernetes.io/projected/ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9-kube-api-access-nbnsv\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:36.353340 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.353309    2579 generic.go:356] "Generic (PLEG): container finished" podID="ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9" containerID="c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45" exitCode=0
Jul 24 12:59:36.353781 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.353352    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5"
Jul 24 12:59:36.353781 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.353397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5" event={"ID":"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9","Type":"ContainerDied","Data":"c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45"}
Jul 24 12:59:36.353781 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.353450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mcqm5" event={"ID":"ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9","Type":"ContainerDied","Data":"d84597611d0c30c81ce7d285cc800e92ae012ba52e7a0855f8d22dc214f40e06"}
Jul 24 12:59:36.353781 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.353482    2579 scope.go:117] "RemoveContainer" containerID="c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45"
Jul 24 12:59:36.362371 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.362354    2579 scope.go:117] "RemoveContainer" containerID="c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45"
Jul 24 12:59:36.362626 ip-10-0-141-29 kubenswrapper[2579]: E0724 12:59:36.362609    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45\": container with ID starting with c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45 not found: ID does not exist" containerID="c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45"
Jul 24 12:59:36.362663 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.362635    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45"} err="failed to get container status \"c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45\": rpc error: code = NotFound desc = could not find container \"c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45\": container with ID starting with c68b32b3861aa1eb83d3597ae590a75e36bb519682e02541dcacbd0a1c880b45 not found: ID does not exist"
Jul 24 12:59:36.375742 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.375717    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mcqm5"]
Jul 24 12:59:36.380636 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.380618    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mcqm5"]
Jul 24 12:59:36.561550 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:36.561516    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9" path="/var/lib/kubelet/pods/ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9/volumes"
Jul 24 12:59:40.000231 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:40.000198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:40.000626 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:40.000283    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:40.004715 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:40.004689    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:40.375473 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:40.375396    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f5c8d5d86-khztv"
Jul 24 12:59:40.431298 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:40.431267    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765676c455-srm8t"]
Jul 24 12:59:42.676274 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:42.676237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:42.676740 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:42.676302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:42.698227 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:42.698203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:43.407993 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:43.407948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7fbm6"
Jul 24 12:59:47.617736 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.617698    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"]
Jul 24 12:59:47.618127 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.618056    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9" containerName="registry-server"
Jul 24 12:59:47.618127 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.618067    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9" containerName="registry-server"
Jul 24 12:59:47.618127 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.618117    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae309bc1-0a8c-4cc7-b4bd-58c4fc59ece9" containerName="registry-server"
Jul 24 12:59:47.621269 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.621253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.624134 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.624096    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvm6g\""
Jul 24 12:59:47.631170 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.631148    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"]
Jul 24 12:59:47.722566 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.722526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.722751 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.722583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.722751 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.722608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82wc\" (UniqueName: \"kubernetes.io/projected/4282687f-1d59-4141-ba82-e597b9885c80-kube-api-access-s82wc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.823986 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.823915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.824142 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.824003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.824142 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.824027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82wc\" (UniqueName: \"kubernetes.io/projected/4282687f-1d59-4141-ba82-e597b9885c80-kube-api-access-s82wc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.824347 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.824330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.824383 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.824355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.834146 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.834120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82wc\" (UniqueName: \"kubernetes.io/projected/4282687f-1d59-4141-ba82-e597b9885c80-kube-api-access-s82wc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:47.930354 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:47.930307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:48.006917 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.006885    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"]
Jul 24 12:59:48.011462 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.011438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.025722 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.025692    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"]
Jul 24 12:59:48.025912 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.025884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.026067 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.025943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26758\" (UniqueName: \"kubernetes.io/projected/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-kube-api-access-26758\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.026067 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.025986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.072749 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.072658    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"]
Jul 24 12:59:48.075046 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:59:48.075018    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4282687f_1d59_4141_ba82_e597b9885c80.slice/crio-df863020e46ac659b77d35ae5551d293589c3b3d715958742663efd8ef4c4ff9 WatchSource:0}: Error finding container df863020e46ac659b77d35ae5551d293589c3b3d715958742663efd8ef4c4ff9: Status 404 returned error can't find the container with id df863020e46ac659b77d35ae5551d293589c3b3d715958742663efd8ef4c4ff9
Jul 24 12:59:48.126558 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.126531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26758\" (UniqueName: \"kubernetes.io/projected/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-kube-api-access-26758\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.126686 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.126574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.126748 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.126691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.127082 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.127063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.127234 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.127090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.136510 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.136487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26758\" (UniqueName: \"kubernetes.io/projected/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-kube-api-access-26758\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.324880 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.324790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:48.404666 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.404634    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"]
Jul 24 12:59:48.409035 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.409011    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.412188 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.412024    2579 generic.go:356] "Generic (PLEG): container finished" podID="4282687f-1d59-4141-ba82-e597b9885c80" containerID="e0b5b16190f4ee7d2c058d59d4c104b35cb80bb50e248a020bec99941272e220" exitCode=0
Jul 24 12:59:48.412188 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.412094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl" event={"ID":"4282687f-1d59-4141-ba82-e597b9885c80","Type":"ContainerDied","Data":"e0b5b16190f4ee7d2c058d59d4c104b35cb80bb50e248a020bec99941272e220"}
Jul 24 12:59:48.412188 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.412182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl" event={"ID":"4282687f-1d59-4141-ba82-e597b9885c80","Type":"ContainerStarted","Data":"df863020e46ac659b77d35ae5551d293589c3b3d715958742663efd8ef4c4ff9"}
Jul 24 12:59:48.424161 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.424135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"]
Jul 24 12:59:48.431044 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.430986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.431179 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.431089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.431179 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.431137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg6rt\" (UniqueName: \"kubernetes.io/projected/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-kube-api-access-sg6rt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.465170 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.465146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"]
Jul 24 12:59:48.466608 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:59:48.466583    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b88c58_6b97_4a6f_82ac_9eb37ce9b5f4.slice/crio-ed81645fa73876638c2bc4ce911e3e0a295bb70cbfbe6426cd96d1f1df29ee6a WatchSource:0}: Error finding container ed81645fa73876638c2bc4ce911e3e0a295bb70cbfbe6426cd96d1f1df29ee6a: Status 404 returned error can't find the container with id ed81645fa73876638c2bc4ce911e3e0a295bb70cbfbe6426cd96d1f1df29ee6a
Jul 24 12:59:48.531578 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.531553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6rt\" (UniqueName: \"kubernetes.io/projected/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-kube-api-access-sg6rt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.531664 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.531643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.531701 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.531689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.532010 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.531993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.532060 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.532043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.547020 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.546976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6rt\" (UniqueName: \"kubernetes.io/projected/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-kube-api-access-sg6rt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.721577 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.721543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:48.855266 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:48.855243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"]
Jul 24 12:59:48.857262 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:59:48.857224    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fdd71d_ae33_4712_a8a7_3aab60aa15d4.slice/crio-ea97f2d58865540284d37650f9bb1185f8b2495905150ab4a15b26200f22d884 WatchSource:0}: Error finding container ea97f2d58865540284d37650f9bb1185f8b2495905150ab4a15b26200f22d884: Status 404 returned error can't find the container with id ea97f2d58865540284d37650f9bb1185f8b2495905150ab4a15b26200f22d884
Jul 24 12:59:49.022784 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.022705    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"]
Jul 24 12:59:49.026112 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.026095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.045046 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.045015    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"]
Jul 24 12:59:49.135938 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.135898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.136091 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.135966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brk2m\" (UniqueName: \"kubernetes.io/projected/e8d7eae2-427a-44d2-989d-7f357e36101d-kube-api-access-brk2m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.136091 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.136044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.236716 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.236680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.236881 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.236788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.236881 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.236820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brk2m\" (UniqueName: \"kubernetes.io/projected/e8d7eae2-427a-44d2-989d-7f357e36101d-kube-api-access-brk2m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.237103 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.237083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.237175 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.237104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.248192 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.248169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brk2m\" (UniqueName: \"kubernetes.io/projected/e8d7eae2-427a-44d2-989d-7f357e36101d-kube-api-access-brk2m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.334834 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.334759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:49.420676 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.420573    2579 generic.go:356] "Generic (PLEG): container finished" podID="4282687f-1d59-4141-ba82-e597b9885c80" containerID="b93d0d78f76617c66d4cd41d8d2f550f76bacc40496356764d319c44ea2bf65e" exitCode=0
Jul 24 12:59:49.420836 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.420737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl" event={"ID":"4282687f-1d59-4141-ba82-e597b9885c80","Type":"ContainerDied","Data":"b93d0d78f76617c66d4cd41d8d2f550f76bacc40496356764d319c44ea2bf65e"}
Jul 24 12:59:49.422770 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.422749    2579 generic.go:356] "Generic (PLEG): container finished" podID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerID="1ee87a9001fb8601ecce0811047f494165f8243c76fbbab172a6b1c9917b9b5e" exitCode=0
Jul 24 12:59:49.422876 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.422807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj" event={"ID":"17fdd71d-ae33-4712-a8a7-3aab60aa15d4","Type":"ContainerDied","Data":"1ee87a9001fb8601ecce0811047f494165f8243c76fbbab172a6b1c9917b9b5e"}
Jul 24 12:59:49.422876 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.422835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj" event={"ID":"17fdd71d-ae33-4712-a8a7-3aab60aa15d4","Type":"ContainerStarted","Data":"ea97f2d58865540284d37650f9bb1185f8b2495905150ab4a15b26200f22d884"}
Jul 24 12:59:49.425231 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.424679    2579 generic.go:356] "Generic (PLEG): container finished" podID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerID="b2ae03c6038489586adf2bfa6001229d64e9d5146b13c10890b785e348eb7fb0" exitCode=0
Jul 24 12:59:49.425231 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.424715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp" event={"ID":"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4","Type":"ContainerDied","Data":"b2ae03c6038489586adf2bfa6001229d64e9d5146b13c10890b785e348eb7fb0"}
Jul 24 12:59:49.425231 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.424750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp" event={"ID":"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4","Type":"ContainerStarted","Data":"ed81645fa73876638c2bc4ce911e3e0a295bb70cbfbe6426cd96d1f1df29ee6a"}
Jul 24 12:59:49.483745 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:49.483716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"]
Jul 24 12:59:49.484885 ip-10-0-141-29 kubenswrapper[2579]: W0724 12:59:49.484860    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d7eae2_427a_44d2_989d_7f357e36101d.slice/crio-98de64506b58edcf01e0ee477d88f60003d1a4dd9773eb6d007e280a757381b5 WatchSource:0}: Error finding container 98de64506b58edcf01e0ee477d88f60003d1a4dd9773eb6d007e280a757381b5: Status 404 returned error can't find the container with id 98de64506b58edcf01e0ee477d88f60003d1a4dd9773eb6d007e280a757381b5
Jul 24 12:59:50.431611 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:50.431568    2579 generic.go:356] "Generic (PLEG): container finished" podID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerID="88237ab0bb366109e0e1c424973675b57abe356ca314b3082c7bdd2d983de173" exitCode=0
Jul 24 12:59:50.432046 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:50.431657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp" event={"ID":"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4","Type":"ContainerDied","Data":"88237ab0bb366109e0e1c424973675b57abe356ca314b3082c7bdd2d983de173"}
Jul 24 12:59:50.433247 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:50.433215    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerID="678aadbdd70c93f38fd15c030579c5908a4b66875ee0c90bd54bbdb0de76cd3c" exitCode=0
Jul 24 12:59:50.433306 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:50.433278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq" event={"ID":"e8d7eae2-427a-44d2-989d-7f357e36101d","Type":"ContainerDied","Data":"678aadbdd70c93f38fd15c030579c5908a4b66875ee0c90bd54bbdb0de76cd3c"}
Jul 24 12:59:50.433351 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:50.433329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq" event={"ID":"e8d7eae2-427a-44d2-989d-7f357e36101d","Type":"ContainerStarted","Data":"98de64506b58edcf01e0ee477d88f60003d1a4dd9773eb6d007e280a757381b5"}
Jul 24 12:59:50.435511 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:50.435487    2579 generic.go:356] "Generic (PLEG): container finished" podID="4282687f-1d59-4141-ba82-e597b9885c80" containerID="f86c5cca1365eae3982e9454d601f84bbdc387abc71cd78ac0c468246a5672d9" exitCode=0
Jul 24 12:59:50.435623 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:50.435511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl" event={"ID":"4282687f-1d59-4141-ba82-e597b9885c80","Type":"ContainerDied","Data":"f86c5cca1365eae3982e9454d601f84bbdc387abc71cd78ac0c468246a5672d9"}
Jul 24 12:59:51.442660 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.442630    2579 generic.go:356] "Generic (PLEG): container finished" podID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerID="b14128484a7aefad6d7fb3b794599f17e6b63bfe70ae4eb9cd518a75cf2b6ef9" exitCode=0
Jul 24 12:59:51.443060 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.442722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj" event={"ID":"17fdd71d-ae33-4712-a8a7-3aab60aa15d4","Type":"ContainerDied","Data":"b14128484a7aefad6d7fb3b794599f17e6b63bfe70ae4eb9cd518a75cf2b6ef9"}
Jul 24 12:59:51.444878 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.444856    2579 generic.go:356] "Generic (PLEG): container finished" podID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerID="001471ea9db9daaf37d99872b8424159e13ac68422a05d69417ba1edd51fa3cc" exitCode=0
Jul 24 12:59:51.444984 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.444942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp" event={"ID":"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4","Type":"ContainerDied","Data":"001471ea9db9daaf37d99872b8424159e13ac68422a05d69417ba1edd51fa3cc"}
Jul 24 12:59:51.446759 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.446737    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerID="aa7b8b21e97d40c42c0d1d187e2a0db64039786de8cc1aa78f54dfe2dbcdec08" exitCode=0
Jul 24 12:59:51.446846 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.446826    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq" event={"ID":"e8d7eae2-427a-44d2-989d-7f357e36101d","Type":"ContainerDied","Data":"aa7b8b21e97d40c42c0d1d187e2a0db64039786de8cc1aa78f54dfe2dbcdec08"}
Jul 24 12:59:51.573024 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.572964    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:51.658234 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.658207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-util\") pod \"4282687f-1d59-4141-ba82-e597b9885c80\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") "
Jul 24 12:59:51.658362 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.658238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-bundle\") pod \"4282687f-1d59-4141-ba82-e597b9885c80\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") "
Jul 24 12:59:51.658362 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.658280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s82wc\" (UniqueName: \"kubernetes.io/projected/4282687f-1d59-4141-ba82-e597b9885c80-kube-api-access-s82wc\") pod \"4282687f-1d59-4141-ba82-e597b9885c80\" (UID: \"4282687f-1d59-4141-ba82-e597b9885c80\") "
Jul 24 12:59:51.658910 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.658887    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-bundle" (OuterVolumeSpecName: "bundle") pod "4282687f-1d59-4141-ba82-e597b9885c80" (UID: "4282687f-1d59-4141-ba82-e597b9885c80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:59:51.660495 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.660475    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4282687f-1d59-4141-ba82-e597b9885c80-kube-api-access-s82wc" (OuterVolumeSpecName: "kube-api-access-s82wc") pod "4282687f-1d59-4141-ba82-e597b9885c80" (UID: "4282687f-1d59-4141-ba82-e597b9885c80"). InnerVolumeSpecName "kube-api-access-s82wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:59:51.663977 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.663942    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-util" (OuterVolumeSpecName: "util") pod "4282687f-1d59-4141-ba82-e597b9885c80" (UID: "4282687f-1d59-4141-ba82-e597b9885c80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:59:51.759242 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.759208    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s82wc\" (UniqueName: \"kubernetes.io/projected/4282687f-1d59-4141-ba82-e597b9885c80-kube-api-access-s82wc\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:51.759242 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.759236    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-util\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:51.759242 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:51.759244    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4282687f-1d59-4141-ba82-e597b9885c80-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:52.453822 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.453785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl" event={"ID":"4282687f-1d59-4141-ba82-e597b9885c80","Type":"ContainerDied","Data":"df863020e46ac659b77d35ae5551d293589c3b3d715958742663efd8ef4c4ff9"}
Jul 24 12:59:52.454302 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.453838    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df863020e46ac659b77d35ae5551d293589c3b3d715958742663efd8ef4c4ff9"
Jul 24 12:59:52.454302 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.453798    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl"
Jul 24 12:59:52.462445 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.462417    2579 generic.go:356] "Generic (PLEG): container finished" podID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerID="836a94e328625d643c0102d83ea67f59474a64255fa5e6e76731a7dae3c919ca" exitCode=0
Jul 24 12:59:52.462595 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.462489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj" event={"ID":"17fdd71d-ae33-4712-a8a7-3aab60aa15d4","Type":"ContainerDied","Data":"836a94e328625d643c0102d83ea67f59474a64255fa5e6e76731a7dae3c919ca"}
Jul 24 12:59:52.464967 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.464694    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerID="7a781f093d0bf85cda0e3f500c776dbe0a83f3a6d09a05ece6187f67f326e415" exitCode=0
Jul 24 12:59:52.464967 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.464824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq" event={"ID":"e8d7eae2-427a-44d2-989d-7f357e36101d","Type":"ContainerDied","Data":"7a781f093d0bf85cda0e3f500c776dbe0a83f3a6d09a05ece6187f67f326e415"}
Jul 24 12:59:52.600214 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.600192    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:52.668233 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.668206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-util\") pod \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") "
Jul 24 12:59:52.668396 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.668257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-bundle\") pod \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") "
Jul 24 12:59:52.668396 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.668275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26758\" (UniqueName: \"kubernetes.io/projected/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-kube-api-access-26758\") pod \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\" (UID: \"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4\") "
Jul 24 12:59:52.668737 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.668715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-bundle" (OuterVolumeSpecName: "bundle") pod "33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" (UID: "33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:59:52.670515 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.670495    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-kube-api-access-26758" (OuterVolumeSpecName: "kube-api-access-26758") pod "33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" (UID: "33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4"). InnerVolumeSpecName "kube-api-access-26758". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:59:52.673979 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.673913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-util" (OuterVolumeSpecName: "util") pod "33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" (UID: "33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:59:52.769549 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.769463    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-util\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:52.769549 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.769493    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:52.769549 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:52.769502    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26758\" (UniqueName: \"kubernetes.io/projected/33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4-kube-api-access-26758\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:53.471047 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.471013    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp"
Jul 24 12:59:53.471440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.471049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp" event={"ID":"33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4","Type":"ContainerDied","Data":"ed81645fa73876638c2bc4ce911e3e0a295bb70cbfbe6426cd96d1f1df29ee6a"}
Jul 24 12:59:53.471440 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.471096    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed81645fa73876638c2bc4ce911e3e0a295bb70cbfbe6426cd96d1f1df29ee6a"
Jul 24 12:59:53.643185 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.643166    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:53.645238 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.645210    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 12:59:53.677600 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.677572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brk2m\" (UniqueName: \"kubernetes.io/projected/e8d7eae2-427a-44d2-989d-7f357e36101d-kube-api-access-brk2m\") pod \"e8d7eae2-427a-44d2-989d-7f357e36101d\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") "
Jul 24 12:59:53.677732 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.677655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg6rt\" (UniqueName: \"kubernetes.io/projected/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-kube-api-access-sg6rt\") pod \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") "
Jul 24 12:59:53.677732 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.677726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-bundle\") pod \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") "
Jul 24 12:59:53.677840 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.677767    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-util\") pod \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\" (UID: \"17fdd71d-ae33-4712-a8a7-3aab60aa15d4\") "
Jul 24 12:59:53.677840 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.677793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-bundle\") pod \"e8d7eae2-427a-44d2-989d-7f357e36101d\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") "
Jul 24 12:59:53.677964 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.677843    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-util\") pod \"e8d7eae2-427a-44d2-989d-7f357e36101d\" (UID: \"e8d7eae2-427a-44d2-989d-7f357e36101d\") "
Jul 24 12:59:53.678406 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.678373    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-bundle" (OuterVolumeSpecName: "bundle") pod "e8d7eae2-427a-44d2-989d-7f357e36101d" (UID: "e8d7eae2-427a-44d2-989d-7f357e36101d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:59:53.678495 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.678423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-bundle" (OuterVolumeSpecName: "bundle") pod "17fdd71d-ae33-4712-a8a7-3aab60aa15d4" (UID: "17fdd71d-ae33-4712-a8a7-3aab60aa15d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:59:53.680275 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.680248    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d7eae2-427a-44d2-989d-7f357e36101d-kube-api-access-brk2m" (OuterVolumeSpecName: "kube-api-access-brk2m") pod "e8d7eae2-427a-44d2-989d-7f357e36101d" (UID: "e8d7eae2-427a-44d2-989d-7f357e36101d"). InnerVolumeSpecName "kube-api-access-brk2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:59:53.680480 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.680465    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-kube-api-access-sg6rt" (OuterVolumeSpecName: "kube-api-access-sg6rt") pod "17fdd71d-ae33-4712-a8a7-3aab60aa15d4" (UID: "17fdd71d-ae33-4712-a8a7-3aab60aa15d4"). InnerVolumeSpecName "kube-api-access-sg6rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:59:53.684254 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.684215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-util" (OuterVolumeSpecName: "util") pod "17fdd71d-ae33-4712-a8a7-3aab60aa15d4" (UID: "17fdd71d-ae33-4712-a8a7-3aab60aa15d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:59:53.686447 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.686423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-util" (OuterVolumeSpecName: "util") pod "e8d7eae2-427a-44d2-989d-7f357e36101d" (UID: "e8d7eae2-427a-44d2-989d-7f357e36101d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:59:53.778942 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.778836    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-util\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:53.778942 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.778870    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:53.778942 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.778883    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8d7eae2-427a-44d2-989d-7f357e36101d-util\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:53.778942 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.778894    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brk2m\" (UniqueName: \"kubernetes.io/projected/e8d7eae2-427a-44d2-989d-7f357e36101d-kube-api-access-brk2m\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:53.778942 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.778909    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg6rt\" (UniqueName: \"kubernetes.io/projected/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-kube-api-access-sg6rt\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:53.778942 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:53.778936    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17fdd71d-ae33-4712-a8a7-3aab60aa15d4-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 12:59:54.478123 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:54.478094    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj"
Jul 24 12:59:54.478585 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:54.478087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj" event={"ID":"17fdd71d-ae33-4712-a8a7-3aab60aa15d4","Type":"ContainerDied","Data":"ea97f2d58865540284d37650f9bb1185f8b2495905150ab4a15b26200f22d884"}
Jul 24 12:59:54.478585 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:54.478207    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea97f2d58865540284d37650f9bb1185f8b2495905150ab4a15b26200f22d884"
Jul 24 12:59:54.479850 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:54.479825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq" event={"ID":"e8d7eae2-427a-44d2-989d-7f357e36101d","Type":"ContainerDied","Data":"98de64506b58edcf01e0ee477d88f60003d1a4dd9773eb6d007e280a757381b5"}
Jul 24 12:59:54.479983 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:54.479856    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98de64506b58edcf01e0ee477d88f60003d1a4dd9773eb6d007e280a757381b5"
Jul 24 12:59:54.479983 ip-10-0-141-29 kubenswrapper[2579]: I0724 12:59:54.479864    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq"
Jul 24 13:00:05.450421 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.450376    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-765676c455-srm8t" podUID="40c9b00f-a03d-42e5-a53d-fea52b8dfb08" containerName="console" containerID="cri-o://b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e" gracePeriod=15
Jul 24 13:00:05.699961 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.699938    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765676c455-srm8t"
Jul 24 13:00:05.789766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.789681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-config\") pod \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") "
Jul 24 13:00:05.789766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.789734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-serving-cert\") pod \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") "
Jul 24 13:00:05.789766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.789764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-trusted-ca-bundle\") pod \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") "
Jul 24 13:00:05.790072 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.789794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sk44\" (UniqueName: \"kubernetes.io/projected/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-kube-api-access-2sk44\") pod \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") "
Jul 24 13:00:05.790072 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.789835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-oauth-config\") pod \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") "
Jul 24 13:00:05.790072 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.789965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-oauth-serving-cert\") pod \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") "
Jul 24 13:00:05.790072 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.790024    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-service-ca\") pod \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\" (UID: \"40c9b00f-a03d-42e5-a53d-fea52b8dfb08\") "
Jul 24 13:00:05.790268 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.790213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-config" (OuterVolumeSpecName: "console-config") pod "40c9b00f-a03d-42e5-a53d-fea52b8dfb08" (UID: "40c9b00f-a03d-42e5-a53d-fea52b8dfb08"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:00:05.790724 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.790398    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:05.790724 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.790597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "40c9b00f-a03d-42e5-a53d-fea52b8dfb08" (UID: "40c9b00f-a03d-42e5-a53d-fea52b8dfb08"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:00:05.790901 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.790797    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "40c9b00f-a03d-42e5-a53d-fea52b8dfb08" (UID: "40c9b00f-a03d-42e5-a53d-fea52b8dfb08"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:00:05.791029 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.790998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-service-ca" (OuterVolumeSpecName: "service-ca") pod "40c9b00f-a03d-42e5-a53d-fea52b8dfb08" (UID: "40c9b00f-a03d-42e5-a53d-fea52b8dfb08"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:00:05.792866 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.792815    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "40c9b00f-a03d-42e5-a53d-fea52b8dfb08" (UID: "40c9b00f-a03d-42e5-a53d-fea52b8dfb08"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:00:05.793009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.792989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "40c9b00f-a03d-42e5-a53d-fea52b8dfb08" (UID: "40c9b00f-a03d-42e5-a53d-fea52b8dfb08"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:00:05.793140 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.793115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-kube-api-access-2sk44" (OuterVolumeSpecName: "kube-api-access-2sk44") pod "40c9b00f-a03d-42e5-a53d-fea52b8dfb08" (UID: "40c9b00f-a03d-42e5-a53d-fea52b8dfb08"). InnerVolumeSpecName "kube-api-access-2sk44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:00:05.891074 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.891042    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-serving-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:05.891074 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.891065    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-trusted-ca-bundle\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:05.891250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.891074    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sk44\" (UniqueName: \"kubernetes.io/projected/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-kube-api-access-2sk44\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:05.891250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.891149    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-console-oauth-config\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:05.891250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.891161    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-oauth-serving-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:05.891250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:05.891170    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40c9b00f-a03d-42e5-a53d-fea52b8dfb08-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:06.540501 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.540464    2579 generic.go:356] "Generic (PLEG): container finished" podID="40c9b00f-a03d-42e5-a53d-fea52b8dfb08" containerID="b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e" exitCode=0
Jul 24 13:00:06.540501 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.540498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765676c455-srm8t" event={"ID":"40c9b00f-a03d-42e5-a53d-fea52b8dfb08","Type":"ContainerDied","Data":"b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e"}
Jul 24 13:00:06.541035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.540523    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765676c455-srm8t"
Jul 24 13:00:06.541035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.540563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765676c455-srm8t" event={"ID":"40c9b00f-a03d-42e5-a53d-fea52b8dfb08","Type":"ContainerDied","Data":"2d4aa1b3eec0e0411f62fa903336aaaa68d94202c95dc53a2b105c6ff11cb71f"}
Jul 24 13:00:06.541035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.540599    2579 scope.go:117] "RemoveContainer" containerID="b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e"
Jul 24 13:00:06.550902 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.550889    2579 scope.go:117] "RemoveContainer" containerID="b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e"
Jul 24 13:00:06.551177 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:06.551159    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e\": container with ID starting with b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e not found: ID does not exist" containerID="b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e"
Jul 24 13:00:06.551224 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.551185    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e"} err="failed to get container status \"b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e\": rpc error: code = NotFound desc = could not find container \"b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e\": container with ID starting with b8b6c3463decac8df58d2f384f7ad2b888d3d1c43425b130d59fe94d61c3e49e not found: ID does not exist"
Jul 24 13:00:06.570538 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.567524    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765676c455-srm8t"]
Jul 24 13:00:06.577342 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:06.577317    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765676c455-srm8t"]
Jul 24 13:00:08.361722 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.361691    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"]
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362034    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerName="pull"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362045    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerName="pull"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362056    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c9b00f-a03d-42e5-a53d-fea52b8dfb08" containerName="console"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362061    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c9b00f-a03d-42e5-a53d-fea52b8dfb08" containerName="console"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362066    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4282687f-1d59-4141-ba82-e597b9885c80" containerName="extract"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362075    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4282687f-1d59-4141-ba82-e597b9885c80" containerName="extract"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362089    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerName="util"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362095    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerName="util"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362101    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerName="pull"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362105    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerName="pull"
Jul 24 13:00:08.362105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362111    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4282687f-1d59-4141-ba82-e597b9885c80" containerName="util"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362116    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4282687f-1d59-4141-ba82-e597b9885c80" containerName="util"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362121    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerName="pull"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362126    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerName="pull"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362133    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerName="util"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362138    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerName="util"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362145    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4282687f-1d59-4141-ba82-e597b9885c80" containerName="pull"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362150    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4282687f-1d59-4141-ba82-e597b9885c80" containerName="pull"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362159    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362167    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362177    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362182    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362187    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerName="util"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362191    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerName="util"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362197    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362202    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362254    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362261    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fdd71d-ae33-4712-a8a7-3aab60aa15d4" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362268    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d7eae2-427a-44d2-989d-7f357e36101d" containerName="extract"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362273    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c9b00f-a03d-42e5-a53d-fea52b8dfb08" containerName="console"
Jul 24 13:00:08.362417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.362280    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4282687f-1d59-4141-ba82-e597b9885c80" containerName="extract"
Jul 24 13:00:08.369856 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.369837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"
Jul 24 13:00:08.376253 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.376232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-c2726\""
Jul 24 13:00:08.380685 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.380660    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"]
Jul 24 13:00:08.516052 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.516017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt96\" (UniqueName: \"kubernetes.io/projected/1b399331-e78b-4b11-ba9c-cf460defbacf-kube-api-access-gzt96\") pod \"authorino-operator-75cc8d5cb7-78qz2\" (UID: \"1b399331-e78b-4b11-ba9c-cf460defbacf\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"
Jul 24 13:00:08.562551 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.562521    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c9b00f-a03d-42e5-a53d-fea52b8dfb08" path="/var/lib/kubelet/pods/40c9b00f-a03d-42e5-a53d-fea52b8dfb08/volumes"
Jul 24 13:00:08.617150 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.617072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt96\" (UniqueName: \"kubernetes.io/projected/1b399331-e78b-4b11-ba9c-cf460defbacf-kube-api-access-gzt96\") pod \"authorino-operator-75cc8d5cb7-78qz2\" (UID: \"1b399331-e78b-4b11-ba9c-cf460defbacf\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"
Jul 24 13:00:08.632844 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.632819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt96\" (UniqueName: \"kubernetes.io/projected/1b399331-e78b-4b11-ba9c-cf460defbacf-kube-api-access-gzt96\") pod \"authorino-operator-75cc8d5cb7-78qz2\" (UID: \"1b399331-e78b-4b11-ba9c-cf460defbacf\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"
Jul 24 13:00:08.679036 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.679009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"
Jul 24 13:00:08.820435 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:08.820401    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"]
Jul 24 13:00:08.825666 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:00:08.825643    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b399331_e78b_4b11_ba9c_cf460defbacf.slice/crio-3884a3f3dc0ae116668fba920ef8d69bcbe875f18dbf0c03be3af671b16c71cc WatchSource:0}: Error finding container 3884a3f3dc0ae116668fba920ef8d69bcbe875f18dbf0c03be3af671b16c71cc: Status 404 returned error can't find the container with id 3884a3f3dc0ae116668fba920ef8d69bcbe875f18dbf0c03be3af671b16c71cc
Jul 24 13:00:09.558301 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:09.558249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2" event={"ID":"1b399331-e78b-4b11-ba9c-cf460defbacf","Type":"ContainerStarted","Data":"3884a3f3dc0ae116668fba920ef8d69bcbe875f18dbf0c03be3af671b16c71cc"}
Jul 24 13:00:11.570114 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.570077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2" event={"ID":"1b399331-e78b-4b11-ba9c-cf460defbacf","Type":"ContainerStarted","Data":"9faf6a41fa1c3e67d3eeded9ed287688faee801e8d1f42bc4b3d69616d279f45"}
Jul 24 13:00:11.570419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.570172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"
Jul 24 13:00:11.591870 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.591820    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2" podStartSLOduration=0.92461025 podStartE2EDuration="3.591801496s" totalImagesPullingTime="2.667191246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:00:11.591320118 +0000 UTC m=+435.580638827" watchObservedRunningTime="2026-07-24 13:00:11.591801496 +0000 UTC m=+435.581120192"
Jul 24 13:00:11.664543 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.664512    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"]
Jul 24 13:00:11.667952 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.667917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"
Jul 24 13:00:11.671588 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.671566    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 24 13:00:11.671813 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.671798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9jgg7\""
Jul 24 13:00:11.682456 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.682429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"]
Jul 24 13:00:11.844697 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.844607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkkf\" (UniqueName: \"kubernetes.io/projected/11abd561-d41e-48ae-b913-cce05671a8d4-kube-api-access-5kkkf\") pod \"dns-operator-controller-manager-864dd78c4c-w46b8\" (UID: \"11abd561-d41e-48ae-b913-cce05671a8d4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"
Jul 24 13:00:11.946073 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.946035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkkf\" (UniqueName: \"kubernetes.io/projected/11abd561-d41e-48ae-b913-cce05671a8d4-kube-api-access-5kkkf\") pod \"dns-operator-controller-manager-864dd78c4c-w46b8\" (UID: \"11abd561-d41e-48ae-b913-cce05671a8d4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"
Jul 24 13:00:11.959561 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.959536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkkf\" (UniqueName: \"kubernetes.io/projected/11abd561-d41e-48ae-b913-cce05671a8d4-kube-api-access-5kkkf\") pod \"dns-operator-controller-manager-864dd78c4c-w46b8\" (UID: \"11abd561-d41e-48ae-b913-cce05671a8d4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"
Jul 24 13:00:11.977515 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:11.977494    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"
Jul 24 13:00:12.116787 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:12.116672    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"]
Jul 24 13:00:12.118604 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:00:12.118564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11abd561_d41e_48ae_b913_cce05671a8d4.slice/crio-6fa8e8e674220f12dc86badc5ebd044326ec3cb31a11b1ce95bfb5bb839ed5c7 WatchSource:0}: Error finding container 6fa8e8e674220f12dc86badc5ebd044326ec3cb31a11b1ce95bfb5bb839ed5c7: Status 404 returned error can't find the container with id 6fa8e8e674220f12dc86badc5ebd044326ec3cb31a11b1ce95bfb5bb839ed5c7
Jul 24 13:00:12.575433 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:12.575399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8" event={"ID":"11abd561-d41e-48ae-b913-cce05671a8d4","Type":"ContainerStarted","Data":"6fa8e8e674220f12dc86badc5ebd044326ec3cb31a11b1ce95bfb5bb839ed5c7"}
Jul 24 13:00:14.588210 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:14.588126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8" event={"ID":"11abd561-d41e-48ae-b913-cce05671a8d4","Type":"ContainerStarted","Data":"484e251684db6d2d19d352e0231a643b71df3d377096f9cfd3b1e62b7548de48"}
Jul 24 13:00:14.588577 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:14.588265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"
Jul 24 13:00:14.619527 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:14.619478    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8" podStartSLOduration=1.575820097 podStartE2EDuration="3.619464576s" totalImagesPullingTime="2.043644479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:00:14.616771491 +0000 UTC m=+438.606090188" watchObservedRunningTime="2026-07-24 13:00:14.619464576 +0000 UTC m=+438.608783338"
Jul 24 13:00:16.726373 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.726337    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"]
Jul 24 13:00:16.729667 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.729645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:16.732729 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.732708    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7djvc\""
Jul 24 13:00:16.745984 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.745956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"]
Jul 24 13:00:16.888792 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.888694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/919acb42-77ad-4788-975a-3202d27a1b4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" (UID: \"919acb42-77ad-4788-975a-3202d27a1b4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:16.888987 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.888833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5btn\" (UniqueName: \"kubernetes.io/projected/919acb42-77ad-4788-975a-3202d27a1b4e-kube-api-access-k5btn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" (UID: \"919acb42-77ad-4788-975a-3202d27a1b4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:16.989435 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.989341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5btn\" (UniqueName: \"kubernetes.io/projected/919acb42-77ad-4788-975a-3202d27a1b4e-kube-api-access-k5btn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" (UID: \"919acb42-77ad-4788-975a-3202d27a1b4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:16.989435 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.989396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/919acb42-77ad-4788-975a-3202d27a1b4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" (UID: \"919acb42-77ad-4788-975a-3202d27a1b4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:16.989753 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:16.989731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/919acb42-77ad-4788-975a-3202d27a1b4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" (UID: \"919acb42-77ad-4788-975a-3202d27a1b4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:17.003732 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:17.003700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5btn\" (UniqueName: \"kubernetes.io/projected/919acb42-77ad-4788-975a-3202d27a1b4e-kube-api-access-k5btn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" (UID: \"919acb42-77ad-4788-975a-3202d27a1b4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:17.039471 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:17.039432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:17.208249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:17.208205    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"]
Jul 24 13:00:17.212817 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:00:17.212789    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919acb42_77ad_4788_975a_3202d27a1b4e.slice/crio-29142e7d308c225b4dda6c62d841ea2e39d59b73e2700fc7721d02c82c76c388 WatchSource:0}: Error finding container 29142e7d308c225b4dda6c62d841ea2e39d59b73e2700fc7721d02c82c76c388: Status 404 returned error can't find the container with id 29142e7d308c225b4dda6c62d841ea2e39d59b73e2700fc7721d02c82c76c388
Jul 24 13:00:17.603401 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:17.603366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq" event={"ID":"919acb42-77ad-4788-975a-3202d27a1b4e","Type":"ContainerStarted","Data":"29142e7d308c225b4dda6c62d841ea2e39d59b73e2700fc7721d02c82c76c388"}
Jul 24 13:00:22.578437 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:22.578408    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-78qz2"
Jul 24 13:00:23.642187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:23.642147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq" event={"ID":"919acb42-77ad-4788-975a-3202d27a1b4e","Type":"ContainerStarted","Data":"14a83382372be872b4f0c7da4964c83afac61add9bd47255415d1d1a9e1b92ee"}
Jul 24 13:00:23.642586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:23.642328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:23.677134 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:23.677074    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq" podStartSLOduration=1.988627315 podStartE2EDuration="7.677056474s" totalImagesPullingTime="5.688429159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:00:23.67491281 +0000 UTC m=+447.664231505" watchObservedRunningTime="2026-07-24 13:00:23.677056474 +0000 UTC m=+447.666375172"
Jul 24 13:00:25.595653 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.595625    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-w46b8"
Jul 24 13:00:25.753069 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.753030    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"]
Jul 24 13:00:25.756583 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.756561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:25.759507 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.759485    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 24 13:00:25.759608 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.759512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 24 13:00:25.759608 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.759497    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvm6g\""
Jul 24 13:00:25.763511 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.763487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wff\" (UniqueName: \"kubernetes.io/projected/0f5fef98-a7fd-4e52-b282-4329ace17916-kube-api-access-m7wff\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:25.763620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.763601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f5fef98-a7fd-4e52-b282-4329ace17916-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:25.763690 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.763658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f5fef98-a7fd-4e52-b282-4329ace17916-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:25.768341 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.768320    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"]
Jul 24 13:00:25.864883 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.864794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f5fef98-a7fd-4e52-b282-4329ace17916-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:25.864883 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.864854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f5fef98-a7fd-4e52-b282-4329ace17916-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:25.865127 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.864901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wff\" (UniqueName: \"kubernetes.io/projected/0f5fef98-a7fd-4e52-b282-4329ace17916-kube-api-access-m7wff\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:25.865127 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:25.864983    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 24 13:00:25.865127 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:25.865056    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5fef98-a7fd-4e52-b282-4329ace17916-plugin-serving-cert podName:0f5fef98-a7fd-4e52-b282-4329ace17916 nodeName:}" failed. No retries permitted until 2026-07-24 13:00:26.365038337 +0000 UTC m=+450.354357013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0f5fef98-a7fd-4e52-b282-4329ace17916-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-z2gjj" (UID: "0f5fef98-a7fd-4e52-b282-4329ace17916") : secret "plugin-serving-cert" not found
Jul 24 13:00:25.865669 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.865648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f5fef98-a7fd-4e52-b282-4329ace17916-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:25.886022 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:25.885988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wff\" (UniqueName: \"kubernetes.io/projected/0f5fef98-a7fd-4e52-b282-4329ace17916-kube-api-access-m7wff\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:26.368118 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:26.368072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f5fef98-a7fd-4e52-b282-4329ace17916-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:26.370819 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:26.370782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f5fef98-a7fd-4e52-b282-4329ace17916-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z2gjj\" (UID: \"0f5fef98-a7fd-4e52-b282-4329ace17916\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:26.665664 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:26.665625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"
Jul 24 13:00:26.806060 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:26.806035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj"]
Jul 24 13:00:26.807791 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:00:26.807767    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f5fef98_a7fd_4e52_b282_4329ace17916.slice/crio-664dead76f05cb0113af4eca21e30ffd4cf270af199d5cb1d4ba4b9eeae0a4cf WatchSource:0}: Error finding container 664dead76f05cb0113af4eca21e30ffd4cf270af199d5cb1d4ba4b9eeae0a4cf: Status 404 returned error can't find the container with id 664dead76f05cb0113af4eca21e30ffd4cf270af199d5cb1d4ba4b9eeae0a4cf
Jul 24 13:00:27.665094 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:27.665058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj" event={"ID":"0f5fef98-a7fd-4e52-b282-4329ace17916","Type":"ContainerStarted","Data":"664dead76f05cb0113af4eca21e30ffd4cf270af199d5cb1d4ba4b9eeae0a4cf"}
Jul 24 13:00:34.650975 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:34.650941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:36.714575 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.714538    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"]
Jul 24 13:00:36.715212 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.714794    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" containerName="manager" containerID="cri-o://14a83382372be872b4f0c7da4964c83afac61add9bd47255415d1d1a9e1b92ee" gracePeriod=2
Jul 24 13:00:36.723676 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.723640    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"]
Jul 24 13:00:36.753964 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.753909    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"]
Jul 24 13:00:36.754458 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.754437    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" containerName="manager"
Jul 24 13:00:36.754547 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.754461    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" containerName="manager"
Jul 24 13:00:36.754602 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.754556    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" containerName="manager"
Jul 24 13:00:36.759173 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.759152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:36.764398 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:36.764370    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" is forbidden: User \"system:node:ip-10-0-141-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-29.ec2.internal' and this object" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:36.797516 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.797481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"]
Jul 24 13:00:36.802009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.801982    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"]
Jul 24 13:00:36.814948 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.814898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:36.825647 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.823526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"]
Jul 24 13:00:36.854094 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:36.854052    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" is forbidden: User \"system:node:ip-10-0-141-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-29.ec2.internal' and this object" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:36.871629 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.871594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc92ef37-8ad3-4513-b6aa-7273e47ec298-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b8bz5\" (UID: \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:36.871820 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.871655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9435e139-b131-46a2-9016-8d5086d75dc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-drgjx\" (UID: \"9435e139-b131-46a2-9016-8d5086d75dc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:36.871820 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.871747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2lg\" (UniqueName: \"kubernetes.io/projected/cc92ef37-8ad3-4513-b6aa-7273e47ec298-kube-api-access-lh2lg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b8bz5\" (UID: \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:36.871948 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.871826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzlmk\" (UniqueName: \"kubernetes.io/projected/9435e139-b131-46a2-9016-8d5086d75dc8-kube-api-access-dzlmk\") pod \"kuadrant-operator-controller-manager-86878c769f-drgjx\" (UID: \"9435e139-b131-46a2-9016-8d5086d75dc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:36.973029 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.972941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc92ef37-8ad3-4513-b6aa-7273e47ec298-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b8bz5\" (UID: \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:36.973029 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.973003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9435e139-b131-46a2-9016-8d5086d75dc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-drgjx\" (UID: \"9435e139-b131-46a2-9016-8d5086d75dc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:36.973260 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.973058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2lg\" (UniqueName: \"kubernetes.io/projected/cc92ef37-8ad3-4513-b6aa-7273e47ec298-kube-api-access-lh2lg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b8bz5\" (UID: \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:36.973260 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.973117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzlmk\" (UniqueName: \"kubernetes.io/projected/9435e139-b131-46a2-9016-8d5086d75dc8-kube-api-access-dzlmk\") pod \"kuadrant-operator-controller-manager-86878c769f-drgjx\" (UID: \"9435e139-b131-46a2-9016-8d5086d75dc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:36.973368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.973353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc92ef37-8ad3-4513-b6aa-7273e47ec298-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b8bz5\" (UID: \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:36.973417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.973369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9435e139-b131-46a2-9016-8d5086d75dc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-drgjx\" (UID: \"9435e139-b131-46a2-9016-8d5086d75dc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:36.985797 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.985770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2lg\" (UniqueName: \"kubernetes.io/projected/cc92ef37-8ad3-4513-b6aa-7273e47ec298-kube-api-access-lh2lg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b8bz5\" (UID: \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:36.986048 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:36.986022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzlmk\" (UniqueName: \"kubernetes.io/projected/9435e139-b131-46a2-9016-8d5086d75dc8-kube-api-access-dzlmk\") pod \"kuadrant-operator-controller-manager-86878c769f-drgjx\" (UID: \"9435e139-b131-46a2-9016-8d5086d75dc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:37.105496 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:37.105459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:37.130948 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:37.130894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:49.805455 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:49.805403    2579 generic.go:356] "Generic (PLEG): container finished" podID="919acb42-77ad-4788-975a-3202d27a1b4e" containerID="14a83382372be872b4f0c7da4964c83afac61add9bd47255415d1d1a9e1b92ee" exitCode=0
Jul 24 13:00:49.975666 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:49.975638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"]
Jul 24 13:00:49.996193 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:49.996167    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"]
Jul 24 13:00:50.146601 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:00:50.146025    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc92ef37_8ad3_4513_b6aa_7273e47ec298.slice/crio-437783ce48b2b61d2116736522b6add0e84b5543f4c30312a6f354e8e091467b WatchSource:0}: Error finding container 437783ce48b2b61d2116736522b6add0e84b5543f4c30312a6f354e8e091467b: Status 404 returned error can't find the container with id 437783ce48b2b61d2116736522b6add0e84b5543f4c30312a6f354e8e091467b
Jul 24 13:00:50.188450 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.188425    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:50.190879 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:50.190852    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" is forbidden: User \"system:node:ip-10-0-141-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-29.ec2.internal' and this object" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:50.292380 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.292350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5btn\" (UniqueName: \"kubernetes.io/projected/919acb42-77ad-4788-975a-3202d27a1b4e-kube-api-access-k5btn\") pod \"919acb42-77ad-4788-975a-3202d27a1b4e\" (UID: \"919acb42-77ad-4788-975a-3202d27a1b4e\") "
Jul 24 13:00:50.292525 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.292421    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/919acb42-77ad-4788-975a-3202d27a1b4e-extensions-socket-volume\") pod \"919acb42-77ad-4788-975a-3202d27a1b4e\" (UID: \"919acb42-77ad-4788-975a-3202d27a1b4e\") "
Jul 24 13:00:50.292953 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.292906    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919acb42-77ad-4788-975a-3202d27a1b4e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "919acb42-77ad-4788-975a-3202d27a1b4e" (UID: "919acb42-77ad-4788-975a-3202d27a1b4e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:00:50.294601 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.294573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919acb42-77ad-4788-975a-3202d27a1b4e-kube-api-access-k5btn" (OuterVolumeSpecName: "kube-api-access-k5btn") pod "919acb42-77ad-4788-975a-3202d27a1b4e" (UID: "919acb42-77ad-4788-975a-3202d27a1b4e"). InnerVolumeSpecName "kube-api-access-k5btn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:00:50.393343 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.393255    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5btn\" (UniqueName: \"kubernetes.io/projected/919acb42-77ad-4788-975a-3202d27a1b4e-kube-api-access-k5btn\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:50.393343 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.393284    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/919acb42-77ad-4788-975a-3202d27a1b4e-extensions-socket-volume\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:00:50.569029 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.568992    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" path="/var/lib/kubelet/pods/919acb42-77ad-4788-975a-3202d27a1b4e/volumes"
Jul 24 13:00:50.812330 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.812309    2579 scope.go:117] "RemoveContainer" containerID="14a83382372be872b4f0c7da4964c83afac61add9bd47255415d1d1a9e1b92ee"
Jul 24 13:00:50.812330 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.812317    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:50.814110 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.814083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj" event={"ID":"0f5fef98-a7fd-4e52-b282-4329ace17916","Type":"ContainerStarted","Data":"1706b45aa11e2ba464e84109dc9247b354f1ab7e5dedc6c2a289ca1759f13053"}
Jul 24 13:00:50.815517 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:50.815490    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" is forbidden: User \"system:node:ip-10-0-141-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-29.ec2.internal' and this object" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:50.816032 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.816002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx" event={"ID":"9435e139-b131-46a2-9016-8d5086d75dc8","Type":"ContainerStarted","Data":"eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682"}
Jul 24 13:00:50.816375 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.816062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx" event={"ID":"9435e139-b131-46a2-9016-8d5086d75dc8","Type":"ContainerStarted","Data":"8eca8e191de81710fe3471ae931a76de871355f3583eaf06a905e743a379a763"}
Jul 24 13:00:50.816375 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.816151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:00:50.817626 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.817606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5" event={"ID":"cc92ef37-8ad3-4513-b6aa-7273e47ec298","Type":"ContainerStarted","Data":"be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542"}
Jul 24 13:00:50.817713 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.817649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5" event={"ID":"cc92ef37-8ad3-4513-b6aa-7273e47ec298","Type":"ContainerStarted","Data":"437783ce48b2b61d2116736522b6add0e84b5543f4c30312a6f354e8e091467b"}
Jul 24 13:00:50.817713 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.817697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:00:50.837979 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.837903    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z2gjj" podStartSLOduration=2.428943141 podStartE2EDuration="25.837890398s" totalImagesPullingTime="23.408947257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:00:50.837700284 +0000 UTC m=+474.827018980" watchObservedRunningTime="2026-07-24 13:00:50.837890398 +0000 UTC m=+474.827209094"
Jul 24 13:00:50.840063 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:50.840026    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" is forbidden: User \"system:node:ip-10-0-141-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-29.ec2.internal' and this object" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:50.870005 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.869944    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5" podStartSLOduration=14.869908348 podStartE2EDuration="14.869908348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:00:50.867198042 +0000 UTC m=+474.856516752" watchObservedRunningTime="2026-07-24 13:00:50.869908348 +0000 UTC m=+474.859227044"
Jul 24 13:00:50.895623 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:00:50.895562    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nlftq\" is forbidden: User \"system:node:ip-10-0-141-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-29.ec2.internal' and this object" podUID="919acb42-77ad-4788-975a-3202d27a1b4e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlftq"
Jul 24 13:00:50.896020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:00:50.895972    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx" podStartSLOduration=14.895956826 podStartE2EDuration="14.895956826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:00:50.892730527 +0000 UTC m=+474.882049221" watchObservedRunningTime="2026-07-24 13:00:50.895956826 +0000 UTC m=+474.885275522"
Jul 24 13:01:01.825784 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:01.825743    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:01:01.826295 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:01.825841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:01:01.923607 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:01.923573    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"]
Jul 24 13:01:01.923815 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:01.923787    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5" podUID="cc92ef37-8ad3-4513-b6aa-7273e47ec298" containerName="manager" containerID="cri-o://be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542" gracePeriod=10
Jul 24 13:01:02.173883 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.173857    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:01:02.301087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.301058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh2lg\" (UniqueName: \"kubernetes.io/projected/cc92ef37-8ad3-4513-b6aa-7273e47ec298-kube-api-access-lh2lg\") pod \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\" (UID: \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\") "
Jul 24 13:01:02.301257 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.301129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc92ef37-8ad3-4513-b6aa-7273e47ec298-extensions-socket-volume\") pod \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\" (UID: \"cc92ef37-8ad3-4513-b6aa-7273e47ec298\") "
Jul 24 13:01:02.301533 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.301509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc92ef37-8ad3-4513-b6aa-7273e47ec298-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cc92ef37-8ad3-4513-b6aa-7273e47ec298" (UID: "cc92ef37-8ad3-4513-b6aa-7273e47ec298"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:01:02.303359 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.303327    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc92ef37-8ad3-4513-b6aa-7273e47ec298-kube-api-access-lh2lg" (OuterVolumeSpecName: "kube-api-access-lh2lg") pod "cc92ef37-8ad3-4513-b6aa-7273e47ec298" (UID: "cc92ef37-8ad3-4513-b6aa-7273e47ec298"). InnerVolumeSpecName "kube-api-access-lh2lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:01:02.402047 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.401912    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2lg\" (UniqueName: \"kubernetes.io/projected/cc92ef37-8ad3-4513-b6aa-7273e47ec298-kube-api-access-lh2lg\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:01:02.402047 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.401989    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc92ef37-8ad3-4513-b6aa-7273e47ec298-extensions-socket-volume\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:01:02.882217 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.882175    2579 generic.go:356] "Generic (PLEG): container finished" podID="cc92ef37-8ad3-4513-b6aa-7273e47ec298" containerID="be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542" exitCode=0
Jul 24 13:01:02.882625 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.882227    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"
Jul 24 13:01:02.882625 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.882238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5" event={"ID":"cc92ef37-8ad3-4513-b6aa-7273e47ec298","Type":"ContainerDied","Data":"be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542"}
Jul 24 13:01:02.882625 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.882293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5" event={"ID":"cc92ef37-8ad3-4513-b6aa-7273e47ec298","Type":"ContainerDied","Data":"437783ce48b2b61d2116736522b6add0e84b5543f4c30312a6f354e8e091467b"}
Jul 24 13:01:02.882625 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.882323    2579 scope.go:117] "RemoveContainer" containerID="be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542"
Jul 24 13:01:02.891247 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.891224    2579 scope.go:117] "RemoveContainer" containerID="be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542"
Jul 24 13:01:02.891557 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:01:02.891535    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542\": container with ID starting with be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542 not found: ID does not exist" containerID="be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542"
Jul 24 13:01:02.891653 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.891564    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542"} err="failed to get container status \"be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542\": rpc error: code = NotFound desc = could not find container \"be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542\": container with ID starting with be98676961caa92f720dd8108b3b5c354ecd6b262ba64c74aa5bbda84d66f542 not found: ID does not exist"
Jul 24 13:01:02.904673 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.904640    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"]
Jul 24 13:01:02.907859 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:02.907830    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b8bz5"]
Jul 24 13:01:04.563401 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:04.563360    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc92ef37-8ad3-4513-b6aa-7273e47ec298" path="/var/lib/kubelet/pods/cc92ef37-8ad3-4513-b6aa-7273e47ec298/volumes"
Jul 24 13:01:18.481741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.481705    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"]
Jul 24 13:01:18.482279 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.482081    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc92ef37-8ad3-4513-b6aa-7273e47ec298" containerName="manager"
Jul 24 13:01:18.482279 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.482093    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc92ef37-8ad3-4513-b6aa-7273e47ec298" containerName="manager"
Jul 24 13:01:18.482279 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.482166    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc92ef37-8ad3-4513-b6aa-7273e47ec298" containerName="manager"
Jul 24 13:01:18.497361 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.497306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.500149 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.500119    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r2tpk\""
Jul 24 13:01:18.500357 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.500338    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"]
Jul 24 13:01:18.536836 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.536799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2q8r\" (UniqueName: \"kubernetes.io/projected/8797c594-88cf-45b9-8ac9-2028500f389d-kube-api-access-q2q8r\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.537051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.536849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.537051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.536906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.537051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.536975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.537051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.537021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8797c594-88cf-45b9-8ac9-2028500f389d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.537051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.537044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8797c594-88cf-45b9-8ac9-2028500f389d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.537289 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.537071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8797c594-88cf-45b9-8ac9-2028500f389d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.537289 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.537107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.537289 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.537173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638154 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638154 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2q8r\" (UniqueName: \"kubernetes.io/projected/8797c594-88cf-45b9-8ac9-2028500f389d-kube-api-access-q2q8r\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8797c594-88cf-45b9-8ac9-2028500f389d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8797c594-88cf-45b9-8ac9-2028500f389d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8797c594-88cf-45b9-8ac9-2028500f389d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638786 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638786 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638786 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.638957 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.638792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.639184 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.639166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8797c594-88cf-45b9-8ac9-2028500f389d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.640799 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.640777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8797c594-88cf-45b9-8ac9-2028500f389d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.641063 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.641041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8797c594-88cf-45b9-8ac9-2028500f389d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.649211 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.649182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8797c594-88cf-45b9-8ac9-2028500f389d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.649330 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.649314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2q8r\" (UniqueName: \"kubernetes.io/projected/8797c594-88cf-45b9-8ac9-2028500f389d-kube-api-access-q2q8r\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkr97\" (UID: \"8797c594-88cf-45b9-8ac9-2028500f389d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.809104 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.809013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:18.983193 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.983165    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"]
Jul 24 13:01:18.985210 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:01:18.985182    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8797c594_88cf_45b9_8ac9_2028500f389d.slice/crio-f398ae42c5b9407ad6215a966a13c00ea0e175311c2f02b658991bf5e95c6e64 WatchSource:0}: Error finding container f398ae42c5b9407ad6215a966a13c00ea0e175311c2f02b658991bf5e95c6e64: Status 404 returned error can't find the container with id f398ae42c5b9407ad6215a966a13c00ea0e175311c2f02b658991bf5e95c6e64
Jul 24 13:01:18.987238 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.987213    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:01:18.987315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.987269    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:01:18.987315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:18.987298    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:01:19.973438 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:19.973396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" event={"ID":"8797c594-88cf-45b9-8ac9-2028500f389d","Type":"ContainerStarted","Data":"ab268885748658544168d3cde31aa1a3dd76f6c8f0543ab8199ebd2195793869"}
Jul 24 13:01:19.973960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:19.973464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" event={"ID":"8797c594-88cf-45b9-8ac9-2028500f389d","Type":"ContainerStarted","Data":"f398ae42c5b9407ad6215a966a13c00ea0e175311c2f02b658991bf5e95c6e64"}
Jul 24 13:01:20.000085 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:20.000020    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" podStartSLOduration=1.999996377 podStartE2EDuration="1.999996377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:01:19.99708464 +0000 UTC m=+503.986403338" watchObservedRunningTime="2026-07-24 13:01:19.999996377 +0000 UTC m=+503.989315074"
Jul 24 13:01:20.809802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:20.809771    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:20.814647 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:20.814616    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:20.978680 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:20.978648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:20.979562 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:20.979539    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:01:22.059322 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.059287    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8g89q"]
Jul 24 13:01:22.061837 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.061815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8g89q"
Jul 24 13:01:22.064648 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.064626    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nbhrf\""
Jul 24 13:01:22.072824 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.072798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8g89q"]
Jul 24 13:01:22.171201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.171160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/39ffef52-1d51-4e96-ac67-30c1e9deae0f-kube-api-access-k6zgv\") pod \"authorino-58cd955675-8g89q\" (UID: \"39ffef52-1d51-4e96-ac67-30c1e9deae0f\") " pod="kuadrant-system/authorino-58cd955675-8g89q"
Jul 24 13:01:22.272239 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.272198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/39ffef52-1d51-4e96-ac67-30c1e9deae0f-kube-api-access-k6zgv\") pod \"authorino-58cd955675-8g89q\" (UID: \"39ffef52-1d51-4e96-ac67-30c1e9deae0f\") " pod="kuadrant-system/authorino-58cd955675-8g89q"
Jul 24 13:01:22.282048 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.282020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/39ffef52-1d51-4e96-ac67-30c1e9deae0f-kube-api-access-k6zgv\") pod \"authorino-58cd955675-8g89q\" (UID: \"39ffef52-1d51-4e96-ac67-30c1e9deae0f\") " pod="kuadrant-system/authorino-58cd955675-8g89q"
Jul 24 13:01:22.372331 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.372236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8g89q"
Jul 24 13:01:22.514724 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.514696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8g89q"]
Jul 24 13:01:22.516975 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:01:22.516945    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39ffef52_1d51_4e96_ac67_30c1e9deae0f.slice/crio-bd804060a4c5711ff96531eefd7630c73503e0183720eb370ce46b474ccbb11b WatchSource:0}: Error finding container bd804060a4c5711ff96531eefd7630c73503e0183720eb370ce46b474ccbb11b: Status 404 returned error can't find the container with id bd804060a4c5711ff96531eefd7630c73503e0183720eb370ce46b474ccbb11b
Jul 24 13:01:22.991007 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:22.990971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8g89q" event={"ID":"39ffef52-1d51-4e96-ac67-30c1e9deae0f","Type":"ContainerStarted","Data":"bd804060a4c5711ff96531eefd7630c73503e0183720eb370ce46b474ccbb11b"}
Jul 24 13:01:26.012003 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:26.011957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8g89q" event={"ID":"39ffef52-1d51-4e96-ac67-30c1e9deae0f","Type":"ContainerStarted","Data":"0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87"}
Jul 24 13:01:26.032439 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:26.032330    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8g89q" podStartSLOduration=0.773377186 podStartE2EDuration="4.032313316s" totalImagesPullingTime="3.25893613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:01:26.029614163 +0000 UTC m=+510.018932860" watchObservedRunningTime="2026-07-24 13:01:26.032313316 +0000 UTC m=+510.021632013"
Jul 24 13:01:27.553672 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:27.553624    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8g89q"]
Jul 24 13:01:28.022597 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:28.022525    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8g89q" podUID="39ffef52-1d51-4e96-ac67-30c1e9deae0f" containerName="authorino" containerID="cri-o://0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87" gracePeriod=30
Jul 24 13:01:28.273534 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:28.273457    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8g89q"
Jul 24 13:01:28.334185 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:28.334139    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/39ffef52-1d51-4e96-ac67-30c1e9deae0f-kube-api-access-k6zgv\") pod \"39ffef52-1d51-4e96-ac67-30c1e9deae0f\" (UID: \"39ffef52-1d51-4e96-ac67-30c1e9deae0f\") "
Jul 24 13:01:28.336447 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:28.336417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ffef52-1d51-4e96-ac67-30c1e9deae0f-kube-api-access-k6zgv" (OuterVolumeSpecName: "kube-api-access-k6zgv") pod "39ffef52-1d51-4e96-ac67-30c1e9deae0f" (UID: "39ffef52-1d51-4e96-ac67-30c1e9deae0f"). InnerVolumeSpecName "kube-api-access-k6zgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:01:28.435704 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:28.435661    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/39ffef52-1d51-4e96-ac67-30c1e9deae0f-kube-api-access-k6zgv\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:01:29.028748 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.028710    2579 generic.go:356] "Generic (PLEG): container finished" podID="39ffef52-1d51-4e96-ac67-30c1e9deae0f" containerID="0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87" exitCode=0
Jul 24 13:01:29.029302 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.028756    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8g89q"
Jul 24 13:01:29.029302 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.028800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8g89q" event={"ID":"39ffef52-1d51-4e96-ac67-30c1e9deae0f","Type":"ContainerDied","Data":"0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87"}
Jul 24 13:01:29.029302 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.028844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8g89q" event={"ID":"39ffef52-1d51-4e96-ac67-30c1e9deae0f","Type":"ContainerDied","Data":"bd804060a4c5711ff96531eefd7630c73503e0183720eb370ce46b474ccbb11b"}
Jul 24 13:01:29.029302 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.028864    2579 scope.go:117] "RemoveContainer" containerID="0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87"
Jul 24 13:01:29.037891 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.037860    2579 scope.go:117] "RemoveContainer" containerID="0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87"
Jul 24 13:01:29.038339 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:01:29.038316    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87\": container with ID starting with 0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87 not found: ID does not exist" containerID="0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87"
Jul 24 13:01:29.038424 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.038346    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87"} err="failed to get container status \"0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87\": rpc error: code = NotFound desc = could not find container \"0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87\": container with ID starting with 0e464fa705f36294f7eee4b76b09f62c664035c1eb2a0c58da822aa1631b0a87 not found: ID does not exist"
Jul 24 13:01:29.049531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.049496    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8g89q"]
Jul 24 13:01:29.052840 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:29.052811    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8g89q"]
Jul 24 13:01:30.563190 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:30.563161    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39ffef52-1d51-4e96-ac67-30c1e9deae0f" path="/var/lib/kubelet/pods/39ffef52-1d51-4e96-ac67-30c1e9deae0f/volumes"
Jul 24 13:01:56.988259 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:56.988224    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9d6h6"]
Jul 24 13:01:56.988706 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:56.988580    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ffef52-1d51-4e96-ac67-30c1e9deae0f" containerName="authorino"
Jul 24 13:01:56.988706 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:56.988591    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ffef52-1d51-4e96-ac67-30c1e9deae0f" containerName="authorino"
Jul 24 13:01:56.988706 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:56.988657    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39ffef52-1d51-4e96-ac67-30c1e9deae0f" containerName="authorino"
Jul 24 13:01:56.993442 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:56.993410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9d6h6"
Jul 24 13:01:56.996293 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:56.996263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nbhrf\""
Jul 24 13:01:56.998980 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:56.998955    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9d6h6"]
Jul 24 13:01:57.086560 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.086523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zt6n\" (UniqueName: \"kubernetes.io/projected/f16035e8-fbdd-4850-8d32-1380b549c577-kube-api-access-5zt6n\") pod \"authorino-5d4b67c446-9d6h6\" (UID: \"f16035e8-fbdd-4850-8d32-1380b549c577\") " pod="kuadrant-system/authorino-5d4b67c446-9d6h6"
Jul 24 13:01:57.187787 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.187752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt6n\" (UniqueName: \"kubernetes.io/projected/f16035e8-fbdd-4850-8d32-1380b549c577-kube-api-access-5zt6n\") pod \"authorino-5d4b67c446-9d6h6\" (UID: \"f16035e8-fbdd-4850-8d32-1380b549c577\") " pod="kuadrant-system/authorino-5d4b67c446-9d6h6"
Jul 24 13:01:57.199772 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.199744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt6n\" (UniqueName: \"kubernetes.io/projected/f16035e8-fbdd-4850-8d32-1380b549c577-kube-api-access-5zt6n\") pod \"authorino-5d4b67c446-9d6h6\" (UID: \"f16035e8-fbdd-4850-8d32-1380b549c577\") " pod="kuadrant-system/authorino-5d4b67c446-9d6h6"
Jul 24 13:01:57.251505 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.251410    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9d6h6"]
Jul 24 13:01:57.251686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.251674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9d6h6"
Jul 24 13:01:57.281954 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.281900    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58464f9959-sz8kp"]
Jul 24 13:01:57.285398 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.285377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:01:57.288233 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.288208    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 13:01:57.295749 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.295707    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58464f9959-sz8kp"]
Jul 24 13:01:57.389668 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.389625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdv5j\" (UniqueName: \"kubernetes.io/projected/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-kube-api-access-bdv5j\") pod \"authorino-58464f9959-sz8kp\" (UID: \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\") " pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:01:57.389852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.389709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-tls-cert\") pod \"authorino-58464f9959-sz8kp\" (UID: \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\") " pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:01:57.393257 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.393230    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9d6h6"]
Jul 24 13:01:57.394988 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:01:57.394965    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf16035e8_fbdd_4850_8d32_1380b549c577.slice/crio-08bd2554711ae12cbe3a01fb4ed5a57137ead76192d1e8bcecbef68b1550585f WatchSource:0}: Error finding container 08bd2554711ae12cbe3a01fb4ed5a57137ead76192d1e8bcecbef68b1550585f: Status 404 returned error can't find the container with id 08bd2554711ae12cbe3a01fb4ed5a57137ead76192d1e8bcecbef68b1550585f
Jul 24 13:01:57.491045 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.491003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdv5j\" (UniqueName: \"kubernetes.io/projected/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-kube-api-access-bdv5j\") pod \"authorino-58464f9959-sz8kp\" (UID: \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\") " pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:01:57.491270 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.491091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-tls-cert\") pod \"authorino-58464f9959-sz8kp\" (UID: \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\") " pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:01:57.493811 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.493777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-tls-cert\") pod \"authorino-58464f9959-sz8kp\" (UID: \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\") " pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:01:57.503673 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.503611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdv5j\" (UniqueName: \"kubernetes.io/projected/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-kube-api-access-bdv5j\") pod \"authorino-58464f9959-sz8kp\" (UID: \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\") " pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:01:57.596937 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.596886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:01:57.737987 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:57.737959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58464f9959-sz8kp"]
Jul 24 13:01:57.738857 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:01:57.738839    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c08241c_8b6d_4eb8_8a0f_235b2d80d51a.slice/crio-857599a3c9daef550a87510e948f15b887c77e5c2a3d06618bd0cf2d56febcb3 WatchSource:0}: Error finding container 857599a3c9daef550a87510e948f15b887c77e5c2a3d06618bd0cf2d56febcb3: Status 404 returned error can't find the container with id 857599a3c9daef550a87510e948f15b887c77e5c2a3d06618bd0cf2d56febcb3
Jul 24 13:01:58.183590 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.183549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9d6h6" event={"ID":"f16035e8-fbdd-4850-8d32-1380b549c577","Type":"ContainerStarted","Data":"c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f"}
Jul 24 13:01:58.184090 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.183599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9d6h6" event={"ID":"f16035e8-fbdd-4850-8d32-1380b549c577","Type":"ContainerStarted","Data":"08bd2554711ae12cbe3a01fb4ed5a57137ead76192d1e8bcecbef68b1550585f"}
Jul 24 13:01:58.184090 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.183629    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-9d6h6" podUID="f16035e8-fbdd-4850-8d32-1380b549c577" containerName="authorino" containerID="cri-o://c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f" gracePeriod=30
Jul 24 13:01:58.186898 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.186859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58464f9959-sz8kp" event={"ID":"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a","Type":"ContainerStarted","Data":"9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641"}
Jul 24 13:01:58.187271 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.186910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58464f9959-sz8kp" event={"ID":"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a","Type":"ContainerStarted","Data":"857599a3c9daef550a87510e948f15b887c77e5c2a3d06618bd0cf2d56febcb3"}
Jul 24 13:01:58.206586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.206533    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-9d6h6" podStartSLOduration=1.840246815 podStartE2EDuration="2.206516334s" totalImagesPullingTime="366.269519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:01:58.205180633 +0000 UTC m=+542.194499333" watchObservedRunningTime="2026-07-24 13:01:58.206516334 +0000 UTC m=+542.195835031"
Jul 24 13:01:58.224578 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.224522    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58464f9959-sz8kp" podStartSLOduration=0.920180972 podStartE2EDuration="1.22450514s" totalImagesPullingTime="304.324168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:01:58.221749611 +0000 UTC m=+542.211068308" watchObservedRunningTime="2026-07-24 13:01:58.22450514 +0000 UTC m=+542.213823838"
Jul 24 13:01:58.429036 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.429008    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9d6h6"
Jul 24 13:01:58.601237 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.601147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zt6n\" (UniqueName: \"kubernetes.io/projected/f16035e8-fbdd-4850-8d32-1380b549c577-kube-api-access-5zt6n\") pod \"f16035e8-fbdd-4850-8d32-1380b549c577\" (UID: \"f16035e8-fbdd-4850-8d32-1380b549c577\") "
Jul 24 13:01:58.603416 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.603386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16035e8-fbdd-4850-8d32-1380b549c577-kube-api-access-5zt6n" (OuterVolumeSpecName: "kube-api-access-5zt6n") pod "f16035e8-fbdd-4850-8d32-1380b549c577" (UID: "f16035e8-fbdd-4850-8d32-1380b549c577"). InnerVolumeSpecName "kube-api-access-5zt6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:01:58.703035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:58.703002    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zt6n\" (UniqueName: \"kubernetes.io/projected/f16035e8-fbdd-4850-8d32-1380b549c577-kube-api-access-5zt6n\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:01:59.194351 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.194311    2579 generic.go:356] "Generic (PLEG): container finished" podID="f16035e8-fbdd-4850-8d32-1380b549c577" containerID="c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f" exitCode=0
Jul 24 13:01:59.194799 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.194360    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9d6h6"
Jul 24 13:01:59.194799 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.194397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9d6h6" event={"ID":"f16035e8-fbdd-4850-8d32-1380b549c577","Type":"ContainerDied","Data":"c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f"}
Jul 24 13:01:59.194799 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.194438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9d6h6" event={"ID":"f16035e8-fbdd-4850-8d32-1380b549c577","Type":"ContainerDied","Data":"08bd2554711ae12cbe3a01fb4ed5a57137ead76192d1e8bcecbef68b1550585f"}
Jul 24 13:01:59.195133 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.194455    2579 scope.go:117] "RemoveContainer" containerID="c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f"
Jul 24 13:01:59.204570 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.204552    2579 scope.go:117] "RemoveContainer" containerID="c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f"
Jul 24 13:01:59.204910 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:01:59.204894    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f\": container with ID starting with c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f not found: ID does not exist" containerID="c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f"
Jul 24 13:01:59.204990 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.204917    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f"} err="failed to get container status \"c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f\": rpc error: code = NotFound desc = could not find container \"c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f\": container with ID starting with c7b89a8473ad88ef2e896ddd916abe8ea8c0aeb1eb339140ba9db0fbea07c30f not found: ID does not exist"
Jul 24 13:01:59.242373 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.242304    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9d6h6"]
Jul 24 13:01:59.245191 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:01:59.245162    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9d6h6"]
Jul 24 13:02:00.563273 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:00.563230    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f16035e8-fbdd-4850-8d32-1380b549c577" path="/var/lib/kubelet/pods/f16035e8-fbdd-4850-8d32-1380b549c577/volumes"
Jul 24 13:02:48.458270 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:48.458236    2579 generic.go:356] "Generic (PLEG): container finished" podID="8797c594-88cf-45b9-8ac9-2028500f389d" containerID="ab268885748658544168d3cde31aa1a3dd76f6c8f0543ab8199ebd2195793869" exitCode=0
Jul 24 13:02:48.458713 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:48.458291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" event={"ID":"8797c594-88cf-45b9-8ac9-2028500f389d","Type":"ContainerDied","Data":"ab268885748658544168d3cde31aa1a3dd76f6c8f0543ab8199ebd2195793869"}
Jul 24 13:02:48.458713 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:48.458659    2579 scope.go:117] "RemoveContainer" containerID="ab268885748658544168d3cde31aa1a3dd76f6c8f0543ab8199ebd2195793869"
Jul 24 13:02:48.459772 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:48.459736    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:02:48.459869 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:48.459828    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:02:48.459910 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:48.459877    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:02:49.465408 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:49.465372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" event={"ID":"8797c594-88cf-45b9-8ac9-2028500f389d","Type":"ContainerStarted","Data":"fdf234f4041207d530d05e8cecf8f5a31b057bfa099ea2eb4a21561c25e6bbdb"}
Jul 24 13:02:49.809953 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:49.809832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:02:49.811596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:49.811211    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkr97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 24 13:02:49.811596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:49.811317    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" podUID="8797c594-88cf-45b9-8ac9-2028500f389d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 24 13:02:50.810256 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:50.810211    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkr97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 24 13:02:50.810773 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:50.810281    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" podUID="8797c594-88cf-45b9-8ac9-2028500f389d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 24 13:02:50.979357 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:50.979322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:02:51.810573 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:51.810528    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkr97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 24 13:02:51.811092 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:51.810623    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" podUID="8797c594-88cf-45b9-8ac9-2028500f389d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 24 13:02:52.810346 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:52.810308    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkr97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 24 13:02:52.810666 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:52.810636    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" podUID="8797c594-88cf-45b9-8ac9-2028500f389d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 24 13:02:53.809715 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:53.809675    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkr97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 24 13:02:53.809949 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:53.809752    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97" podUID="8797c594-88cf-45b9-8ac9-2028500f389d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 24 13:02:54.813606 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:54.813577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:02:54.814694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:02:54.814674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkr97"
Jul 24 13:03:24.793736 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.793653    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"]
Jul 24 13:03:24.794187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.794073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16035e8-fbdd-4850-8d32-1380b549c577" containerName="authorino"
Jul 24 13:03:24.794187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.794086    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16035e8-fbdd-4850-8d32-1380b549c577" containerName="authorino"
Jul 24 13:03:24.794187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.794150    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16035e8-fbdd-4850-8d32-1380b549c577" containerName="authorino"
Jul 24 13:03:24.796804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.796788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.800634 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.800610    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 24 13:03:24.800754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.800610    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 13:03:24.800754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.800665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 13:03:24.800754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.800658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mpstm\""
Jul 24 13:03:24.808324 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.808304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"]
Jul 24 13:03:24.852774 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.852740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.852774 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.852779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdv6w\" (UniqueName: \"kubernetes.io/projected/15855414-ee67-4c02-a7be-8ebb109a07e1-kube-api-access-tdv6w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.853011 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.852849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.853011 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.852869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15855414-ee67-4c02-a7be-8ebb109a07e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.853011 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.852954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.853011 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.852973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954467 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954467 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdv6w\" (UniqueName: \"kubernetes.io/projected/15855414-ee67-4c02-a7be-8ebb109a07e1-kube-api-access-tdv6w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954467 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954467 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15855414-ee67-4c02-a7be-8ebb109a07e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.954905 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.954886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.957699 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.957651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15855414-ee67-4c02-a7be-8ebb109a07e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.957840 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.957788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15855414-ee67-4c02-a7be-8ebb109a07e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:24.969684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:24.969659    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdv6w\" (UniqueName: \"kubernetes.io/projected/15855414-ee67-4c02-a7be-8ebb109a07e1-kube-api-access-tdv6w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rd2gf\" (UID: \"15855414-ee67-4c02-a7be-8ebb109a07e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:25.107119 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:25.107027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:25.245079 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:25.245054    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"]
Jul 24 13:03:25.246664 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:03:25.246636    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15855414_ee67_4c02_a7be_8ebb109a07e1.slice/crio-49a011306f09608d5a1bc2749b25eddb1ee8fd70ec1b33f2f2ca653065c5ce56 WatchSource:0}: Error finding container 49a011306f09608d5a1bc2749b25eddb1ee8fd70ec1b33f2f2ca653065c5ce56: Status 404 returned error can't find the container with id 49a011306f09608d5a1bc2749b25eddb1ee8fd70ec1b33f2f2ca653065c5ce56
Jul 24 13:03:25.248055 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:25.248038    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:03:25.655179 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:25.655140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf" event={"ID":"15855414-ee67-4c02-a7be-8ebb109a07e1","Type":"ContainerStarted","Data":"49a011306f09608d5a1bc2749b25eddb1ee8fd70ec1b33f2f2ca653065c5ce56"}
Jul 24 13:03:26.662971 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:26.662917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf" event={"ID":"15855414-ee67-4c02-a7be-8ebb109a07e1","Type":"ContainerStarted","Data":"6883e7675e476a99253d03cf7a69e39a11382a65796b84202ee5ec401d137173"}
Jul 24 13:03:26.687255 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:26.687193    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf" podStartSLOduration=1.697305639 podStartE2EDuration="2.68717517s" totalImagesPullingTime="989.869531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:03:26.68619975 +0000 UTC m=+630.675518447" watchObservedRunningTime="2026-07-24 13:03:26.68717517 +0000 UTC m=+630.676493869"
Jul 24 13:03:35.107888 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:35.107845    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:35.108319 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:35.107958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:35.119964 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:35.119911    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:03:35.726634 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:03:35.726586    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rd2gf"
Jul 24 13:04:43.294164 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.294125    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58464f9959-sz8kp"]
Jul 24 13:04:43.294784 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.294364    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58464f9959-sz8kp" podUID="6c08241c-8b6d-4eb8-8a0f-235b2d80d51a" containerName="authorino" containerID="cri-o://9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641" gracePeriod=30
Jul 24 13:04:43.551423 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.551352    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:04:43.661238 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.661199    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdv5j\" (UniqueName: \"kubernetes.io/projected/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-kube-api-access-bdv5j\") pod \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\" (UID: \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\") "
Jul 24 13:04:43.661238 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.661246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-tls-cert\") pod \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\" (UID: \"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a\") "
Jul 24 13:04:43.663635 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.663598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-kube-api-access-bdv5j" (OuterVolumeSpecName: "kube-api-access-bdv5j") pod "6c08241c-8b6d-4eb8-8a0f-235b2d80d51a" (UID: "6c08241c-8b6d-4eb8-8a0f-235b2d80d51a"). InnerVolumeSpecName "kube-api-access-bdv5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:04:43.672665 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.672629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6c08241c-8b6d-4eb8-8a0f-235b2d80d51a" (UID: "6c08241c-8b6d-4eb8-8a0f-235b2d80d51a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:04:43.762662 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.762626    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdv5j\" (UniqueName: \"kubernetes.io/projected/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-kube-api-access-bdv5j\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:04:43.762662 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:43.762657    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a-tls-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:04:44.069555 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.069521    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c08241c-8b6d-4eb8-8a0f-235b2d80d51a" containerID="9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641" exitCode=0
Jul 24 13:04:44.069733 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.069565    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58464f9959-sz8kp"
Jul 24 13:04:44.069733 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.069611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58464f9959-sz8kp" event={"ID":"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a","Type":"ContainerDied","Data":"9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641"}
Jul 24 13:04:44.069733 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.069653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58464f9959-sz8kp" event={"ID":"6c08241c-8b6d-4eb8-8a0f-235b2d80d51a","Type":"ContainerDied","Data":"857599a3c9daef550a87510e948f15b887c77e5c2a3d06618bd0cf2d56febcb3"}
Jul 24 13:04:44.069733 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.069671    2579 scope.go:117] "RemoveContainer" containerID="9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641"
Jul 24 13:04:44.079707 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.079687    2579 scope.go:117] "RemoveContainer" containerID="9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641"
Jul 24 13:04:44.080015 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:04:44.079995    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641\": container with ID starting with 9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641 not found: ID does not exist" containerID="9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641"
Jul 24 13:04:44.080118 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.080025    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641"} err="failed to get container status \"9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641\": rpc error: code = NotFound desc = could not find container \"9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641\": container with ID starting with 9bbd6c0bc8edfc4b8166cbd220ed59f9ac8c289f87cb744d3694923e6329e641 not found: ID does not exist"
Jul 24 13:04:44.095252 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.095215    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58464f9959-sz8kp"]
Jul 24 13:04:44.099662 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.099632    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58464f9959-sz8kp"]
Jul 24 13:04:44.564140 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:04:44.564106    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c08241c-8b6d-4eb8-8a0f-235b2d80d51a" path="/var/lib/kubelet/pods/6c08241c-8b6d-4eb8-8a0f-235b2d80d51a/volumes"
Jul 24 13:06:34.387143 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.387061    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-595b47dc5f-dhvhl"]
Jul 24 13:06:34.387620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.387412    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c08241c-8b6d-4eb8-8a0f-235b2d80d51a" containerName="authorino"
Jul 24 13:06:34.387620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.387428    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c08241c-8b6d-4eb8-8a0f-235b2d80d51a" containerName="authorino"
Jul 24 13:06:34.387620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.387516    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c08241c-8b6d-4eb8-8a0f-235b2d80d51a" containerName="authorino"
Jul 24 13:06:34.390571 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.390539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:34.393159 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.393136    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 13:06:34.394302 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.394281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bdlvk\""
Jul 24 13:06:34.394391 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.394319    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 13:06:34.407361 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.407187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-595b47dc5f-dhvhl"]
Jul 24 13:06:34.440345 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.440315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ac1780a-4bfc-4412-bf28-4e9a3810709c-webhook-cert\") pod \"maas-controller-595b47dc5f-dhvhl\" (UID: \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\") " pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:34.440451 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.440381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mgn\" (UniqueName: \"kubernetes.io/projected/9ac1780a-4bfc-4412-bf28-4e9a3810709c-kube-api-access-m4mgn\") pod \"maas-controller-595b47dc5f-dhvhl\" (UID: \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\") " pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:34.540771 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.540737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mgn\" (UniqueName: \"kubernetes.io/projected/9ac1780a-4bfc-4412-bf28-4e9a3810709c-kube-api-access-m4mgn\") pod \"maas-controller-595b47dc5f-dhvhl\" (UID: \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\") " pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:34.540954 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.540807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ac1780a-4bfc-4412-bf28-4e9a3810709c-webhook-cert\") pod \"maas-controller-595b47dc5f-dhvhl\" (UID: \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\") " pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:34.543413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.543385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ac1780a-4bfc-4412-bf28-4e9a3810709c-webhook-cert\") pod \"maas-controller-595b47dc5f-dhvhl\" (UID: \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\") " pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:34.551854 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.551833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mgn\" (UniqueName: \"kubernetes.io/projected/9ac1780a-4bfc-4412-bf28-4e9a3810709c-kube-api-access-m4mgn\") pod \"maas-controller-595b47dc5f-dhvhl\" (UID: \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\") " pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:34.701880 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.701849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:34.840718 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:34.840685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-595b47dc5f-dhvhl"]
Jul 24 13:06:34.841991 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:06:34.841960    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac1780a_4bfc_4412_bf28_4e9a3810709c.slice/crio-772e3dffe486d25676582b3bc68be3a51595c02ab58006e5950dbed93b142cd4 WatchSource:0}: Error finding container 772e3dffe486d25676582b3bc68be3a51595c02ab58006e5950dbed93b142cd4: Status 404 returned error can't find the container with id 772e3dffe486d25676582b3bc68be3a51595c02ab58006e5950dbed93b142cd4
Jul 24 13:06:35.672031 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:35.671957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-595b47dc5f-dhvhl" event={"ID":"9ac1780a-4bfc-4412-bf28-4e9a3810709c","Type":"ContainerStarted","Data":"772e3dffe486d25676582b3bc68be3a51595c02ab58006e5950dbed93b142cd4"}
Jul 24 13:06:37.686163 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:37.686121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-595b47dc5f-dhvhl" event={"ID":"9ac1780a-4bfc-4412-bf28-4e9a3810709c","Type":"ContainerStarted","Data":"999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f"}
Jul 24 13:06:37.686559 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:37.686315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:37.709728 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:37.709677    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-595b47dc5f-dhvhl" podStartSLOduration=1.418358367 podStartE2EDuration="3.709660448s" totalImagesPullingTime="2.291302081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:06:37.707457443 +0000 UTC m=+821.696776152" watchObservedRunningTime="2026-07-24 13:06:37.709660448 +0000 UTC m=+821.698979146"
Jul 24 13:06:48.694321 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:48.694290    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:53.391303 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.391270    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-595b47dc5f-dhvhl"]
Jul 24 13:06:53.391769 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.391535    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-595b47dc5f-dhvhl" podUID="9ac1780a-4bfc-4412-bf28-4e9a3810709c" containerName="manager" containerID="cri-o://999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f" gracePeriod=30
Jul 24 13:06:53.642895 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.642834    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:53.714019 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.713983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4mgn\" (UniqueName: \"kubernetes.io/projected/9ac1780a-4bfc-4412-bf28-4e9a3810709c-kube-api-access-m4mgn\") pod \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\" (UID: \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\") "
Jul 24 13:06:53.714019 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.714020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ac1780a-4bfc-4412-bf28-4e9a3810709c-webhook-cert\") pod \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\" (UID: \"9ac1780a-4bfc-4412-bf28-4e9a3810709c\") "
Jul 24 13:06:53.716375 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.716349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac1780a-4bfc-4412-bf28-4e9a3810709c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9ac1780a-4bfc-4412-bf28-4e9a3810709c" (UID: "9ac1780a-4bfc-4412-bf28-4e9a3810709c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:06:53.716495 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.716464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac1780a-4bfc-4412-bf28-4e9a3810709c-kube-api-access-m4mgn" (OuterVolumeSpecName: "kube-api-access-m4mgn") pod "9ac1780a-4bfc-4412-bf28-4e9a3810709c" (UID: "9ac1780a-4bfc-4412-bf28-4e9a3810709c"). InnerVolumeSpecName "kube-api-access-m4mgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:06:53.769098 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.769062    2579 generic.go:356] "Generic (PLEG): container finished" podID="9ac1780a-4bfc-4412-bf28-4e9a3810709c" containerID="999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f" exitCode=0
Jul 24 13:06:53.769261 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.769105    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-595b47dc5f-dhvhl"
Jul 24 13:06:53.769261 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.769148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-595b47dc5f-dhvhl" event={"ID":"9ac1780a-4bfc-4412-bf28-4e9a3810709c","Type":"ContainerDied","Data":"999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f"}
Jul 24 13:06:53.769261 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.769198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-595b47dc5f-dhvhl" event={"ID":"9ac1780a-4bfc-4412-bf28-4e9a3810709c","Type":"ContainerDied","Data":"772e3dffe486d25676582b3bc68be3a51595c02ab58006e5950dbed93b142cd4"}
Jul 24 13:06:53.769261 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.769219    2579 scope.go:117] "RemoveContainer" containerID="999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f"
Jul 24 13:06:53.780959 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.780876    2579 scope.go:117] "RemoveContainer" containerID="999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f"
Jul 24 13:06:53.781622 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:06:53.781598    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f\": container with ID starting with 999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f not found: ID does not exist" containerID="999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f"
Jul 24 13:06:53.781780 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.781758    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f"} err="failed to get container status \"999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f\": rpc error: code = NotFound desc = could not find container \"999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f\": container with ID starting with 999e0434540830e7348783cefd82655970cc1d421ddf6d3df2a06875b9c9e77f not found: ID does not exist"
Jul 24 13:06:53.795091 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.795058    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-595b47dc5f-dhvhl"]
Jul 24 13:06:53.800242 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.800191    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-595b47dc5f-dhvhl"]
Jul 24 13:06:53.814897 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.814876    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mgn\" (UniqueName: \"kubernetes.io/projected/9ac1780a-4bfc-4412-bf28-4e9a3810709c-kube-api-access-m4mgn\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:06:53.814897 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:53.814897    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ac1780a-4bfc-4412-bf28-4e9a3810709c-webhook-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:06:54.562717 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.562681    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ac1780a-4bfc-4412-bf28-4e9a3810709c" path="/var/lib/kubelet/pods/9ac1780a-4bfc-4412-bf28-4e9a3810709c/volumes"
Jul 24 13:06:54.677008 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.676970    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-595b47dc5f-7ltrv"]
Jul 24 13:06:54.677358 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.677347    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac1780a-4bfc-4412-bf28-4e9a3810709c" containerName="manager"
Jul 24 13:06:54.677405 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.677360    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac1780a-4bfc-4412-bf28-4e9a3810709c" containerName="manager"
Jul 24 13:06:54.677440 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.677422    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac1780a-4bfc-4412-bf28-4e9a3810709c" containerName="manager"
Jul 24 13:06:54.681838 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.681815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:54.685347 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.685310    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 13:06:54.685347 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.685315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 13:06:54.685554 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.685381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bdlvk\""
Jul 24 13:06:54.691883 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.691859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-595b47dc5f-7ltrv"]
Jul 24 13:06:54.824916 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.824818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqhf6\" (UniqueName: \"kubernetes.io/projected/ae66f370-8468-49d2-95c3-c480312dd778-kube-api-access-rqhf6\") pod \"maas-controller-595b47dc5f-7ltrv\" (UID: \"ae66f370-8468-49d2-95c3-c480312dd778\") " pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:54.824916 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.824884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae66f370-8468-49d2-95c3-c480312dd778-webhook-cert\") pod \"maas-controller-595b47dc5f-7ltrv\" (UID: \"ae66f370-8468-49d2-95c3-c480312dd778\") " pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:54.926112 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.926073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae66f370-8468-49d2-95c3-c480312dd778-webhook-cert\") pod \"maas-controller-595b47dc5f-7ltrv\" (UID: \"ae66f370-8468-49d2-95c3-c480312dd778\") " pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:54.926307 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.926169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqhf6\" (UniqueName: \"kubernetes.io/projected/ae66f370-8468-49d2-95c3-c480312dd778-kube-api-access-rqhf6\") pod \"maas-controller-595b47dc5f-7ltrv\" (UID: \"ae66f370-8468-49d2-95c3-c480312dd778\") " pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:54.928719 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.928692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae66f370-8468-49d2-95c3-c480312dd778-webhook-cert\") pod \"maas-controller-595b47dc5f-7ltrv\" (UID: \"ae66f370-8468-49d2-95c3-c480312dd778\") " pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:54.936823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.936796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqhf6\" (UniqueName: \"kubernetes.io/projected/ae66f370-8468-49d2-95c3-c480312dd778-kube-api-access-rqhf6\") pod \"maas-controller-595b47dc5f-7ltrv\" (UID: \"ae66f370-8468-49d2-95c3-c480312dd778\") " pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:54.994112 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:54.994075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:55.134574 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:55.134552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-595b47dc5f-7ltrv"]
Jul 24 13:06:55.136903 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:06:55.136879    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae66f370_8468_49d2_95c3_c480312dd778.slice/crio-4890ddd73e22c7bd1f5115aab4ca1c7d3650c30a7eb9e1ea1b92958cf101a00a WatchSource:0}: Error finding container 4890ddd73e22c7bd1f5115aab4ca1c7d3650c30a7eb9e1ea1b92958cf101a00a: Status 404 returned error can't find the container with id 4890ddd73e22c7bd1f5115aab4ca1c7d3650c30a7eb9e1ea1b92958cf101a00a
Jul 24 13:06:55.781692 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:55.781659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-595b47dc5f-7ltrv" event={"ID":"ae66f370-8468-49d2-95c3-c480312dd778","Type":"ContainerStarted","Data":"6ab448c0f028bee1498358bc9ab1072fc8c7e3a8d882ce2b7552cf87dea9feb0"}
Jul 24 13:06:55.782120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:55.781735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-595b47dc5f-7ltrv" event={"ID":"ae66f370-8468-49d2-95c3-c480312dd778","Type":"ContainerStarted","Data":"4890ddd73e22c7bd1f5115aab4ca1c7d3650c30a7eb9e1ea1b92958cf101a00a"}
Jul 24 13:06:55.782120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:55.781772    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:06:55.814570 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:06:55.814521    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-595b47dc5f-7ltrv" podStartSLOduration=1.27937827 podStartE2EDuration="1.81450547s" totalImagesPullingTime="535.1272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:06:55.813227821 +0000 UTC m=+839.802546519" watchObservedRunningTime="2026-07-24 13:06:55.81450547 +0000 UTC m=+839.803824165"
Jul 24 13:07:06.789062 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:07:06.789025    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-595b47dc5f-7ltrv"
Jul 24 13:11:43.055641 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:43.055601    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"]
Jul 24 13:11:43.056277 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:43.055867    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx" podUID="9435e139-b131-46a2-9016-8d5086d75dc8" containerName="manager" containerID="cri-o://eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682" gracePeriod=10
Jul 24 13:11:46.512415 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:46.512387    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:11:46.575752 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:46.575661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzlmk\" (UniqueName: \"kubernetes.io/projected/9435e139-b131-46a2-9016-8d5086d75dc8-kube-api-access-dzlmk\") pod \"9435e139-b131-46a2-9016-8d5086d75dc8\" (UID: \"9435e139-b131-46a2-9016-8d5086d75dc8\") "
Jul 24 13:11:46.575952 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:46.575785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9435e139-b131-46a2-9016-8d5086d75dc8-extensions-socket-volume\") pod \"9435e139-b131-46a2-9016-8d5086d75dc8\" (UID: \"9435e139-b131-46a2-9016-8d5086d75dc8\") "
Jul 24 13:11:46.576200 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:46.576169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9435e139-b131-46a2-9016-8d5086d75dc8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9435e139-b131-46a2-9016-8d5086d75dc8" (UID: "9435e139-b131-46a2-9016-8d5086d75dc8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:11:46.577941 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:46.577904    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9435e139-b131-46a2-9016-8d5086d75dc8-kube-api-access-dzlmk" (OuterVolumeSpecName: "kube-api-access-dzlmk") pod "9435e139-b131-46a2-9016-8d5086d75dc8" (UID: "9435e139-b131-46a2-9016-8d5086d75dc8"). InnerVolumeSpecName "kube-api-access-dzlmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:11:46.677277 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:46.677231    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzlmk\" (UniqueName: \"kubernetes.io/projected/9435e139-b131-46a2-9016-8d5086d75dc8-kube-api-access-dzlmk\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:11:46.677277 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:46.677267    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9435e139-b131-46a2-9016-8d5086d75dc8-extensions-socket-volume\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:11:47.312177 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.312141    2579 generic.go:356] "Generic (PLEG): container finished" podID="9435e139-b131-46a2-9016-8d5086d75dc8" containerID="eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682" exitCode=0
Jul 24 13:11:47.312360 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.312186    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"
Jul 24 13:11:47.312360 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.312234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx" event={"ID":"9435e139-b131-46a2-9016-8d5086d75dc8","Type":"ContainerDied","Data":"eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682"}
Jul 24 13:11:47.312360 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.312286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx" event={"ID":"9435e139-b131-46a2-9016-8d5086d75dc8","Type":"ContainerDied","Data":"8eca8e191de81710fe3471ae931a76de871355f3583eaf06a905e743a379a763"}
Jul 24 13:11:47.312360 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.312317    2579 scope.go:117] "RemoveContainer" containerID="eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682"
Jul 24 13:11:47.322737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.322717    2579 scope.go:117] "RemoveContainer" containerID="eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682"
Jul 24 13:11:47.323041 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:11:47.323021    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682\": container with ID starting with eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682 not found: ID does not exist" containerID="eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682"
Jul 24 13:11:47.323110 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.323051    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682"} err="failed to get container status \"eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682\": rpc error: code = NotFound desc = could not find container \"eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682\": container with ID starting with eddb7443bd3c0b3c5756754cb2122946b405f9c9e1602a9ac16c72b7bf1e7682 not found: ID does not exist"
Jul 24 13:11:47.343592 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.343559    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"]
Jul 24 13:11:47.355694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:47.355657    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-drgjx"]
Jul 24 13:11:48.562794 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:11:48.562759    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9435e139-b131-46a2-9016-8d5086d75dc8" path="/var/lib/kubelet/pods/9435e139-b131-46a2-9016-8d5086d75dc8/volumes"
Jul 24 13:12:44.274047 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.274011    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"]
Jul 24 13:12:44.274577 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.274368    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9435e139-b131-46a2-9016-8d5086d75dc8" containerName="manager"
Jul 24 13:12:44.274577 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.274379    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9435e139-b131-46a2-9016-8d5086d75dc8" containerName="manager"
Jul 24 13:12:44.274577 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.274446    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9435e139-b131-46a2-9016-8d5086d75dc8" containerName="manager"
Jul 24 13:12:44.277443 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.277427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:44.281065 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.281044    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7djvc\""
Jul 24 13:12:44.290077 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.290048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"]
Jul 24 13:12:44.380390 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.380351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhm6\" (UniqueName: \"kubernetes.io/projected/e6e9d4b5-e091-4969-9af5-e58ee89540fa-kube-api-access-5hhm6\") pod \"kuadrant-operator-controller-manager-86878c769f-k4vdc\" (UID: \"e6e9d4b5-e091-4969-9af5-e58ee89540fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:44.380390 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.380402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6e9d4b5-e091-4969-9af5-e58ee89540fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k4vdc\" (UID: \"e6e9d4b5-e091-4969-9af5-e58ee89540fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:44.481270 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.481226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhm6\" (UniqueName: \"kubernetes.io/projected/e6e9d4b5-e091-4969-9af5-e58ee89540fa-kube-api-access-5hhm6\") pod \"kuadrant-operator-controller-manager-86878c769f-k4vdc\" (UID: \"e6e9d4b5-e091-4969-9af5-e58ee89540fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:44.481476 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.481283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6e9d4b5-e091-4969-9af5-e58ee89540fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k4vdc\" (UID: \"e6e9d4b5-e091-4969-9af5-e58ee89540fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:44.481669 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.481647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6e9d4b5-e091-4969-9af5-e58ee89540fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k4vdc\" (UID: \"e6e9d4b5-e091-4969-9af5-e58ee89540fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:44.491783 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.491749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhm6\" (UniqueName: \"kubernetes.io/projected/e6e9d4b5-e091-4969-9af5-e58ee89540fa-kube-api-access-5hhm6\") pod \"kuadrant-operator-controller-manager-86878c769f-k4vdc\" (UID: \"e6e9d4b5-e091-4969-9af5-e58ee89540fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:44.587821 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.587723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:44.740026 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:44.739993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"]
Jul 24 13:12:44.742381 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:12:44.742355    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e9d4b5_e091_4969_9af5_e58ee89540fa.slice/crio-b0f310a40308fefd48ff2965a83bafc723a1da2624b927e5ad4d9c0fe2f101dc WatchSource:0}: Error finding container b0f310a40308fefd48ff2965a83bafc723a1da2624b927e5ad4d9c0fe2f101dc: Status 404 returned error can't find the container with id b0f310a40308fefd48ff2965a83bafc723a1da2624b927e5ad4d9c0fe2f101dc
Jul 24 13:12:45.608551 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:45.608517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc" event={"ID":"e6e9d4b5-e091-4969-9af5-e58ee89540fa","Type":"ContainerStarted","Data":"fa44396688795a519751a801da3301dcb106e75e5d522add22efb60fda83121d"}
Jul 24 13:12:45.608967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:45.608569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc" event={"ID":"e6e9d4b5-e091-4969-9af5-e58ee89540fa","Type":"ContainerStarted","Data":"b0f310a40308fefd48ff2965a83bafc723a1da2624b927e5ad4d9c0fe2f101dc"}
Jul 24 13:12:45.608967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:45.608654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:12:45.694451 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:45.694398    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc" podStartSLOduration=1.694381371 podStartE2EDuration="1.694381371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:12:45.688449934 +0000 UTC m=+1189.677768621" watchObservedRunningTime="2026-07-24 13:12:45.694381371 +0000 UTC m=+1189.683700067"
Jul 24 13:12:56.617392 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:12:56.617359    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k4vdc"
Jul 24 13:15:00.153077 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:00.153038    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm"]
Jul 24 13:15:00.156213 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:00.156197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm"
Jul 24 13:15:00.159639 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:00.159621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7xjs\""
Jul 24 13:15:00.165315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:00.165293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm"
Jul 24 13:15:00.211621 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:00.211589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm"]
Jul 24 13:15:00.312694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:00.312668    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm"]
Jul 24 13:15:00.314726 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:15:00.314700    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11444f1_8ab4_4308_9db0_812d803c4ef9.slice/crio-b06f8724c5bf69e083541bde9096470cbd76046e213bb3470062661671c3a9a7 WatchSource:0}: Error finding container b06f8724c5bf69e083541bde9096470cbd76046e213bb3470062661671c3a9a7: Status 404 returned error can't find the container with id b06f8724c5bf69e083541bde9096470cbd76046e213bb3470062661671c3a9a7
Jul 24 13:15:00.316654 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:00.316636    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:15:00.337574 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:00.337533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm" event={"ID":"f11444f1-8ab4-4308-9db0-812d803c4ef9","Type":"ContainerStarted","Data":"b06f8724c5bf69e083541bde9096470cbd76046e213bb3470062661671c3a9a7"}
Jul 24 13:15:03.355129 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:03.355096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm" event={"ID":"f11444f1-8ab4-4308-9db0-812d803c4ef9","Type":"ContainerStarted","Data":"4567cfffec5ee62935d4d82c4437a2eb36691e4c9aa4705101584d7732b0849d"}
Jul 24 13:15:03.373564 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:03.373513    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm" podStartSLOduration=1.057176829 podStartE2EDuration="3.373497716s" totalImagesPullingTime="2.316320887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:15:03.371456279 +0000 UTC m=+1327.360774976" watchObservedRunningTime="2026-07-24 13:15:03.373497716 +0000 UTC m=+1327.362816415"
Jul 24 13:15:04.361542 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:04.361504    2579 generic.go:356] "Generic (PLEG): container finished" podID="f11444f1-8ab4-4308-9db0-812d803c4ef9" containerID="4567cfffec5ee62935d4d82c4437a2eb36691e4c9aa4705101584d7732b0849d" exitCode=0
Jul 24 13:15:04.361967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:04.361590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm" event={"ID":"f11444f1-8ab4-4308-9db0-812d803c4ef9","Type":"ContainerDied","Data":"4567cfffec5ee62935d4d82c4437a2eb36691e4c9aa4705101584d7732b0849d"}
Jul 24 13:15:05.495001 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:05.494978    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm"
Jul 24 13:15:06.372995 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:06.372961    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm"
Jul 24 13:15:06.372995 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:06.372974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748315-wffxm" event={"ID":"f11444f1-8ab4-4308-9db0-812d803c4ef9","Type":"ContainerDied","Data":"b06f8724c5bf69e083541bde9096470cbd76046e213bb3470062661671c3a9a7"}
Jul 24 13:15:06.373215 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:06.373021    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b06f8724c5bf69e083541bde9096470cbd76046e213bb3470062661671c3a9a7"
Jul 24 13:15:35.016273 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.016184    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"]
Jul 24 13:15:35.016843 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.016748    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f11444f1-8ab4-4308-9db0-812d803c4ef9" containerName="cleanup"
Jul 24 13:15:35.016843 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.016769    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f11444f1-8ab4-4308-9db0-812d803c4ef9" containerName="cleanup"
Jul 24 13:15:35.017027 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.016869    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f11444f1-8ab4-4308-9db0-812d803c4ef9" containerName="cleanup"
Jul 24 13:15:35.020036 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.020009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.022637 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.022616    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-64fe3701-kserve-self-signed-certs\""
Jul 24 13:15:35.036541 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.036512    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"]
Jul 24 13:15:35.096462 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.096428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-tmp-dir\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.096462 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.096476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvm5\" (UniqueName: \"kubernetes.io/projected/21da3a7d-876f-471c-b08c-7c57ce107c8e-kube-api-access-jdvm5\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.096737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.096573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-dshm\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.096737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.096618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21da3a7d-876f-471c-b08c-7c57ce107c8e-tls-certs\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.096737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.096647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-home\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.096737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.096680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-model-cache\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.197644 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.197607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-tmp-dir\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.197823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.197657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvm5\" (UniqueName: \"kubernetes.io/projected/21da3a7d-876f-471c-b08c-7c57ce107c8e-kube-api-access-jdvm5\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.197823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.197711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-dshm\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.197823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.197735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21da3a7d-876f-471c-b08c-7c57ce107c8e-tls-certs\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.197823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.197761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-home\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.197823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.197781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-model-cache\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.198117 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.198094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-tmp-dir\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.198169 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.198123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-model-cache\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.198205 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.198192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-home\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.200206 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.200187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-dshm\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.200460 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.200444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21da3a7d-876f-471c-b08c-7c57ce107c8e-tls-certs\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.209436 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.209411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvm5\" (UniqueName: \"kubernetes.io/projected/21da3a7d-876f-471c-b08c-7c57ce107c8e-kube-api-access-jdvm5\") pod \"e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") " pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.328815 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.328720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:35.496159 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.496107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"]
Jul 24 13:15:35.498595 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:15:35.498568    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21da3a7d_876f_471c_b08c_7c57ce107c8e.slice/crio-1ecbd44728b8dbb0c77ca15760b47cd0737de08677592fa8b2390f9cfb698a33 WatchSource:0}: Error finding container 1ecbd44728b8dbb0c77ca15760b47cd0737de08677592fa8b2390f9cfb698a33: Status 404 returned error can't find the container with id 1ecbd44728b8dbb0c77ca15760b47cd0737de08677592fa8b2390f9cfb698a33
Jul 24 13:15:35.524499 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:35.524469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw" event={"ID":"21da3a7d-876f-471c-b08c-7c57ce107c8e","Type":"ContainerStarted","Data":"1ecbd44728b8dbb0c77ca15760b47cd0737de08677592fa8b2390f9cfb698a33"}
Jul 24 13:15:36.531241 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:36.531202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw" event={"ID":"21da3a7d-876f-471c-b08c-7c57ce107c8e","Type":"ContainerStarted","Data":"5ce748f2054a6cc4754c2879716468bbb4488aad7ac7a87d9d866b51cebcc973"}
Jul 24 13:15:36.554019 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:36.553969    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw" podStartSLOduration=2.5539524780000002 podStartE2EDuration="2.553952478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:15:36.55290845 +0000 UTC m=+1360.542227138" watchObservedRunningTime="2026-07-24 13:15:36.553952478 +0000 UTC m=+1360.543271173"
Jul 24 13:15:39.321571 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.321531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"]
Jul 24 13:15:39.326467 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.326445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.329646 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.329624    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-64fe3701-kserve-self-signed-certs\""
Jul 24 13:15:39.342471 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.342446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"]
Jul 24 13:15:39.429780 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.429744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-home\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.429780 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.429781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-dshm\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.430025 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.429842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tls-certs\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.430025 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.429867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-model-cache\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.430025 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.429901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khsbz\" (UniqueName: \"kubernetes.io/projected/62ffa38f-b879-446e-bea3-feb2a50c7ed8-kube-api-access-khsbz\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.430025 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.429964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tmp-dir\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.530524 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.530488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tls-certs\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.530686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.530534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-model-cache\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.530686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.530561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khsbz\" (UniqueName: \"kubernetes.io/projected/62ffa38f-b879-446e-bea3-feb2a50c7ed8-kube-api-access-khsbz\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.530686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.530665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tmp-dir\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.530860 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.530781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-home\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.530860 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.530834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-dshm\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.530987 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.530890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-model-cache\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.531082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.531065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tmp-dir\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.531196 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.531174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-home\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.533022 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.532998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-dshm\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.533269 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.533247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tls-certs\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.541072 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.541046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khsbz\" (UniqueName: \"kubernetes.io/projected/62ffa38f-b879-446e-bea3-feb2a50c7ed8-kube-api-access-khsbz\") pod \"e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") " pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.636196 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.636112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:39.775121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:39.775092    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"]
Jul 24 13:15:39.775513 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:15:39.775488    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62ffa38f_b879_446e_bea3_feb2a50c7ed8.slice/crio-1ba609305b8c489c5f16d29f46b184e8201f353d3bf46c8b5e91777feba6e662 WatchSource:0}: Error finding container 1ba609305b8c489c5f16d29f46b184e8201f353d3bf46c8b5e91777feba6e662: Status 404 returned error can't find the container with id 1ba609305b8c489c5f16d29f46b184e8201f353d3bf46c8b5e91777feba6e662
Jul 24 13:15:40.556137 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:40.556099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5" event={"ID":"62ffa38f-b879-446e-bea3-feb2a50c7ed8","Type":"ContainerStarted","Data":"902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134"}
Jul 24 13:15:40.556137 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:40.556140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5" event={"ID":"62ffa38f-b879-446e-bea3-feb2a50c7ed8","Type":"ContainerStarted","Data":"1ba609305b8c489c5f16d29f46b184e8201f353d3bf46c8b5e91777feba6e662"}
Jul 24 13:15:40.576439 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:40.576392    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5" podStartSLOduration=1.576378799 podStartE2EDuration="1.576378799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:15:40.576107562 +0000 UTC m=+1364.565426258" watchObservedRunningTime="2026-07-24 13:15:40.576378799 +0000 UTC m=+1364.565697495"
Jul 24 13:15:45.329368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:45.329329    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:45.329771 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:45.329405    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:45.341518 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:45.341490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:45.592683 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:45.592596    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:15:49.636818 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:49.636781    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:49.637207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:49.636855    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:49.648597 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:49.648576    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:15:50.621375 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:15:50.621348    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:16:23.681728 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:23.681687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"]
Jul 24 13:16:25.260909 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:25.260873    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"]
Jul 24 13:16:38.702047 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:38.701974    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw" podUID="21da3a7d-876f-471c-b08c-7c57ce107c8e" containerName="main" containerID="cri-o://5ce748f2054a6cc4754c2879716468bbb4488aad7ac7a87d9d866b51cebcc973" gracePeriod=45
Jul 24 13:16:38.879828 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:38.879790    2579 generic.go:356] "Generic (PLEG): container finished" podID="21da3a7d-876f-471c-b08c-7c57ce107c8e" containerID="5ce748f2054a6cc4754c2879716468bbb4488aad7ac7a87d9d866b51cebcc973" exitCode=0
Jul 24 13:16:38.880018 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:38.879854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw" event={"ID":"21da3a7d-876f-471c-b08c-7c57ce107c8e","Type":"ContainerDied","Data":"5ce748f2054a6cc4754c2879716468bbb4488aad7ac7a87d9d866b51cebcc973"}
Jul 24 13:16:38.965443 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:38.965370    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:16:39.051861 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.051823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-tmp-dir\") pod \"21da3a7d-876f-471c-b08c-7c57ce107c8e\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") "
Jul 24 13:16:39.052061 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052003    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-home\") pod \"21da3a7d-876f-471c-b08c-7c57ce107c8e\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") "
Jul 24 13:16:39.052061 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-model-cache\") pod \"21da3a7d-876f-471c-b08c-7c57ce107c8e\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") "
Jul 24 13:16:39.052144 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052087    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdvm5\" (UniqueName: \"kubernetes.io/projected/21da3a7d-876f-471c-b08c-7c57ce107c8e-kube-api-access-jdvm5\") pod \"21da3a7d-876f-471c-b08c-7c57ce107c8e\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") "
Jul 24 13:16:39.052144 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21da3a7d-876f-471c-b08c-7c57ce107c8e-tls-certs\") pod \"21da3a7d-876f-471c-b08c-7c57ce107c8e\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") "
Jul 24 13:16:39.052250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052141    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-dshm\") pod \"21da3a7d-876f-471c-b08c-7c57ce107c8e\" (UID: \"21da3a7d-876f-471c-b08c-7c57ce107c8e\") "
Jul 24 13:16:39.052250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052142    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21da3a7d-876f-471c-b08c-7c57ce107c8e" (UID: "21da3a7d-876f-471c-b08c-7c57ce107c8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:16:39.052362 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052272    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-home" (OuterVolumeSpecName: "home") pod "21da3a7d-876f-471c-b08c-7c57ce107c8e" (UID: "21da3a7d-876f-471c-b08c-7c57ce107c8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:16:39.052362 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052318    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "21da3a7d-876f-471c-b08c-7c57ce107c8e" (UID: "21da3a7d-876f-471c-b08c-7c57ce107c8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:16:39.052466 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052439    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-tmp-dir\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:39.052466 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052452    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-home\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:39.052466 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.052462    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-model-cache\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:39.054537 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.054507    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21da3a7d-876f-471c-b08c-7c57ce107c8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21da3a7d-876f-471c-b08c-7c57ce107c8e" (UID: "21da3a7d-876f-471c-b08c-7c57ce107c8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:16:39.054622 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.054532    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-dshm" (OuterVolumeSpecName: "dshm") pod "21da3a7d-876f-471c-b08c-7c57ce107c8e" (UID: "21da3a7d-876f-471c-b08c-7c57ce107c8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:16:39.054622 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.054590    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21da3a7d-876f-471c-b08c-7c57ce107c8e-kube-api-access-jdvm5" (OuterVolumeSpecName: "kube-api-access-jdvm5") pod "21da3a7d-876f-471c-b08c-7c57ce107c8e" (UID: "21da3a7d-876f-471c-b08c-7c57ce107c8e"). InnerVolumeSpecName "kube-api-access-jdvm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:16:39.153205 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.153170    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvm5\" (UniqueName: \"kubernetes.io/projected/21da3a7d-876f-471c-b08c-7c57ce107c8e-kube-api-access-jdvm5\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:39.153205 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.153199    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21da3a7d-876f-471c-b08c-7c57ce107c8e-tls-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:39.153205 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.153211    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21da3a7d-876f-471c-b08c-7c57ce107c8e-dshm\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:39.887470 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.887415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw" event={"ID":"21da3a7d-876f-471c-b08c-7c57ce107c8e","Type":"ContainerDied","Data":"1ecbd44728b8dbb0c77ca15760b47cd0737de08677592fa8b2390f9cfb698a33"}
Jul 24 13:16:39.888009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.887483    2579 scope.go:117] "RemoveContainer" containerID="5ce748f2054a6cc4754c2879716468bbb4488aad7ac7a87d9d866b51cebcc973"
Jul 24 13:16:39.888009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.887431    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"
Jul 24 13:16:39.923060 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.923025    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"]
Jul 24 13:16:39.925172 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:39.925151    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-64fe3701-kserve-66f6c8cf67-zg5rw"]
Jul 24 13:16:40.279835 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.279774    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5" podUID="62ffa38f-b879-446e-bea3-feb2a50c7ed8" containerName="main" containerID="cri-o://902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134" gracePeriod=45
Jul 24 13:16:40.538382 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.538316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:16:40.563423 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.563399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-dshm\") pod \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") "
Jul 24 13:16:40.563586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.563441    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khsbz\" (UniqueName: \"kubernetes.io/projected/62ffa38f-b879-446e-bea3-feb2a50c7ed8-kube-api-access-khsbz\") pod \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") "
Jul 24 13:16:40.563586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.563484    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tmp-dir\") pod \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") "
Jul 24 13:16:40.563586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.563548    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-home\") pod \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") "
Jul 24 13:16:40.563754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.563699    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-model-cache\") pod \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") "
Jul 24 13:16:40.563754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.563744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tls-certs\") pod \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\" (UID: \"62ffa38f-b879-446e-bea3-feb2a50c7ed8\") "
Jul 24 13:16:40.563856 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.563806    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-home" (OuterVolumeSpecName: "home") pod "62ffa38f-b879-446e-bea3-feb2a50c7ed8" (UID: "62ffa38f-b879-446e-bea3-feb2a50c7ed8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:16:40.563856 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.563834    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62ffa38f-b879-446e-bea3-feb2a50c7ed8" (UID: "62ffa38f-b879-446e-bea3-feb2a50c7ed8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:16:40.564099 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.564049    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-model-cache" (OuterVolumeSpecName: "model-cache") pod "62ffa38f-b879-446e-bea3-feb2a50c7ed8" (UID: "62ffa38f-b879-446e-bea3-feb2a50c7ed8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:16:40.564099 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.564060    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tmp-dir\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:40.564099 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.564100    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-home\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:40.565195 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.565168    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21da3a7d-876f-471c-b08c-7c57ce107c8e" path="/var/lib/kubelet/pods/21da3a7d-876f-471c-b08c-7c57ce107c8e/volumes"
Jul 24 13:16:40.566079 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.566057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62ffa38f-b879-446e-bea3-feb2a50c7ed8" (UID: "62ffa38f-b879-446e-bea3-feb2a50c7ed8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:16:40.566304 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.566289    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-dshm" (OuterVolumeSpecName: "dshm") pod "62ffa38f-b879-446e-bea3-feb2a50c7ed8" (UID: "62ffa38f-b879-446e-bea3-feb2a50c7ed8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:16:40.566375 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.566306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ffa38f-b879-446e-bea3-feb2a50c7ed8-kube-api-access-khsbz" (OuterVolumeSpecName: "kube-api-access-khsbz") pod "62ffa38f-b879-446e-bea3-feb2a50c7ed8" (UID: "62ffa38f-b879-446e-bea3-feb2a50c7ed8"). InnerVolumeSpecName "kube-api-access-khsbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:16:40.664754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.664718    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-model-cache\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:40.664754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.664747    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ffa38f-b879-446e-bea3-feb2a50c7ed8-tls-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:40.664754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.664756    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ffa38f-b879-446e-bea3-feb2a50c7ed8-dshm\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:40.664754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.664764    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khsbz\" (UniqueName: \"kubernetes.io/projected/62ffa38f-b879-446e-bea3-feb2a50c7ed8-kube-api-access-khsbz\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:16:40.896043 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.895942    2579 generic.go:356] "Generic (PLEG): container finished" podID="62ffa38f-b879-446e-bea3-feb2a50c7ed8" containerID="902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134" exitCode=0
Jul 24 13:16:40.896043 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.895984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5" event={"ID":"62ffa38f-b879-446e-bea3-feb2a50c7ed8","Type":"ContainerDied","Data":"902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134"}
Jul 24 13:16:40.896043 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.896026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5" event={"ID":"62ffa38f-b879-446e-bea3-feb2a50c7ed8","Type":"ContainerDied","Data":"1ba609305b8c489c5f16d29f46b184e8201f353d3bf46c8b5e91777feba6e662"}
Jul 24 13:16:40.896043 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.896044    2579 scope.go:117] "RemoveContainer" containerID="902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134"
Jul 24 13:16:40.896563 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.895987    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"
Jul 24 13:16:40.905346 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.905328    2579 scope.go:117] "RemoveContainer" containerID="902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134"
Jul 24 13:16:40.905640 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:16:40.905619    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134\": container with ID starting with 902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134 not found: ID does not exist" containerID="902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134"
Jul 24 13:16:40.905711 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.905651    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134"} err="failed to get container status \"902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134\": rpc error: code = NotFound desc = could not find container \"902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134\": container with ID starting with 902803ddef105b922e532e508dc0b8dd2358feeba6bd05a0936d1f89321e7134 not found: ID does not exist"
Jul 24 13:16:40.921022 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.920989    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"]
Jul 24 13:16:40.925425 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:40.925395    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-64fe3701-kserve-5fbddc78c5-9xtw5"]
Jul 24 13:16:42.563615 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:16:42.563571    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ffa38f-b879-446e-bea3-feb2a50c7ed8" path="/var/lib/kubelet/pods/62ffa38f-b879-446e-bea3-feb2a50c7ed8/volumes"
Jul 24 13:20:09.849684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.849639    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"]
Jul 24 13:20:09.852120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.850182    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21da3a7d-876f-471c-b08c-7c57ce107c8e" containerName="main"
Jul 24 13:20:09.852120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.850197    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="21da3a7d-876f-471c-b08c-7c57ce107c8e" containerName="main"
Jul 24 13:20:09.852120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.850224    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ffa38f-b879-446e-bea3-feb2a50c7ed8" containerName="main"
Jul 24 13:20:09.852120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.850232    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ffa38f-b879-446e-bea3-feb2a50c7ed8" containerName="main"
Jul 24 13:20:09.852120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.850326    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ffa38f-b879-446e-bea3-feb2a50c7ed8" containerName="main"
Jul 24 13:20:09.852120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.850343    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="21da3a7d-876f-471c-b08c-7c57ce107c8e" containerName="main"
Jul 24 13:20:09.853519 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.853503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.856182 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.856161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-acfcca72-openshift-default-dockercfg-zlnkp\""
Jul 24 13:20:09.877359 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.877331    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"]
Jul 24 13:20:09.941609 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-socket\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.941609 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-certs\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.941852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx4w8\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-kube-api-access-jx4w8\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.941852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-data\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.941852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-credential-socket\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.941852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-envoy\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.941852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istiod-ca-cert\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.942103 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-token\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:09.942103 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:09.941965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-podinfo\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043369 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-data\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043369 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-credential-socket\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043623 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-envoy\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043623 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istiod-ca-cert\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043623 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-token\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043623 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-podinfo\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043835 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-socket\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043835 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-certs\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043835 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx4w8\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-kube-api-access-jx4w8\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.043835 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.043749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-credential-socket\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.044096 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.044047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-certs\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.044096 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.044054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-socket\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.044355 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.044329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-data\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.044469 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.044329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istiod-ca-cert\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.045936 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.045903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-envoy\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.046222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.046204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-podinfo\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.052907 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.052885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-token\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.053229 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.053208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx4w8\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-kube-api-access-jx4w8\") pod \"e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") " pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.163580 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.163543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:10.311797 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.311766    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"]
Jul 24 13:20:10.313500 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:20:10.313472    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47e7d9d_5203_4ea2_8f3f_a22abad357e0.slice/crio-85344ea138425c5e1da4faf0859bea5ed2c1fd0fe7f01b7074ecf2e94efa1f6f WatchSource:0}: Error finding container 85344ea138425c5e1da4faf0859bea5ed2c1fd0fe7f01b7074ecf2e94efa1f6f: Status 404 returned error can't find the container with id 85344ea138425c5e1da4faf0859bea5ed2c1fd0fe7f01b7074ecf2e94efa1f6f
Jul 24 13:20:10.315569 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.315535    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:10.315667 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.315628    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:10.315712 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:10.315678    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:11.037802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:11.037763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" event={"ID":"f47e7d9d-5203-4ea2-8f3f-a22abad357e0","Type":"ContainerStarted","Data":"f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e"}
Jul 24 13:20:11.038286 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:11.037816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" event={"ID":"f47e7d9d-5203-4ea2-8f3f-a22abad357e0","Type":"ContainerStarted","Data":"85344ea138425c5e1da4faf0859bea5ed2c1fd0fe7f01b7074ecf2e94efa1f6f"}
Jul 24 13:20:11.067402 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:11.067354    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" podStartSLOduration=2.067337958 podStartE2EDuration="2.067337958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:20:11.0644065 +0000 UTC m=+1635.053725195" watchObservedRunningTime="2026-07-24 13:20:11.067337958 +0000 UTC m=+1635.056656654"
Jul 24 13:20:11.164166 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:11.164129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:11.169050 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:11.169022    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:12.043249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.043213    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:12.044105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.044087    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:12.665741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.665708    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"]
Jul 24 13:20:12.675267 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.675238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:12.678018 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.677988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-acfcca72-serving-cert\""
Jul 24 13:20:12.678324 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.678303    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:20:12.678483 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.678305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:20:12.679122 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.679102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:20:12.679699 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.679674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"]
Jul 24 13:20:12.766986 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.766953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:12.767183 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.767102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1a7464-a447-4757-92bf-021df6eb99bc-openshift-service-ca\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:12.767242 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.767187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctjb\" (UniqueName: \"kubernetes.io/projected/4b1a7464-a447-4757-92bf-021df6eb99bc-kube-api-access-gctjb\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:12.868368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.868333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1a7464-a447-4757-92bf-021df6eb99bc-openshift-service-ca\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:12.868567 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.868386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctjb\" (UniqueName: \"kubernetes.io/projected/4b1a7464-a447-4757-92bf-021df6eb99bc-kube-api-access-gctjb\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:12.868567 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.868427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:12.868567 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:20:12.868558    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-serving-cert: secret "maas-api-e2e-ait-acfcca72-serving-cert" not found
Jul 24 13:20:12.868684 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:20:12.868641    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls podName:4b1a7464-a447-4757-92bf-021df6eb99bc nodeName:}" failed. No retries permitted until 2026-07-24 13:20:13.368620265 +0000 UTC m=+1637.357938939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls") pod "maas-api-e2e-ait-acfcca72-56d8b488cf-656t7" (UID: "4b1a7464-a447-4757-92bf-021df6eb99bc") : secret "maas-api-e2e-ait-acfcca72-serving-cert" not found
Jul 24 13:20:12.868996 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.868975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1a7464-a447-4757-92bf-021df6eb99bc-openshift-service-ca\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:12.880124 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:12.880095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctjb\" (UniqueName: \"kubernetes.io/projected/4b1a7464-a447-4757-92bf-021df6eb99bc-kube-api-access-gctjb\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:13.372857 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.372824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:13.375421 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.375393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls\") pod \"maas-api-e2e-ait-acfcca72-56d8b488cf-656t7\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:13.585962 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.585897    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"]
Jul 24 13:20:13.590068 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.590041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:13.590804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.590078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:13.593181 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.593151    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-acfcca72-dockercfg-9wmgd\""
Jul 24 13:20:13.593181 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.593171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-acfcca72\""
Jul 24 13:20:13.600509 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.600475    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"]
Jul 24 13:20:13.676390 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.676356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e98548b-7fbe-45aa-a928-3ca569f02bb9-plugins-config-volume\") pod \"payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv\" (UID: \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\") " pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:13.676545 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.676402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdlg\" (UniqueName: \"kubernetes.io/projected/0e98548b-7fbe-45aa-a928-3ca569f02bb9-kube-api-access-4kdlg\") pod \"payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv\" (UID: \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\") " pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:13.743505 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.743471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"]
Jul 24 13:20:13.745189 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:20:13.745160    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1a7464_a447_4757_92bf_021df6eb99bc.slice/crio-8b5ee14aadf6b4e1c45bf2ab74ce74d32798ae9508fdae09b4d076dde543fe78 WatchSource:0}: Error finding container 8b5ee14aadf6b4e1c45bf2ab74ce74d32798ae9508fdae09b4d076dde543fe78: Status 404 returned error can't find the container with id 8b5ee14aadf6b4e1c45bf2ab74ce74d32798ae9508fdae09b4d076dde543fe78
Jul 24 13:20:13.746673 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.746653    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:20:13.778009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.777963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e98548b-7fbe-45aa-a928-3ca569f02bb9-plugins-config-volume\") pod \"payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv\" (UID: \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\") " pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:13.778009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.778007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdlg\" (UniqueName: \"kubernetes.io/projected/0e98548b-7fbe-45aa-a928-3ca569f02bb9-kube-api-access-4kdlg\") pod \"payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv\" (UID: \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\") " pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:13.778704 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.778684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e98548b-7fbe-45aa-a928-3ca569f02bb9-plugins-config-volume\") pod \"payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv\" (UID: \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\") " pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:13.787033 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.787006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdlg\" (UniqueName: \"kubernetes.io/projected/0e98548b-7fbe-45aa-a928-3ca569f02bb9-kube-api-access-4kdlg\") pod \"payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv\" (UID: \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\") " pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:13.925888 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:13.925853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:14.054304 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:14.054264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7" event={"ID":"4b1a7464-a447-4757-92bf-021df6eb99bc","Type":"ContainerStarted","Data":"8b5ee14aadf6b4e1c45bf2ab74ce74d32798ae9508fdae09b4d076dde543fe78"}
Jul 24 13:20:14.068248 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:14.068210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"]
Jul 24 13:20:14.069464 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:20:14.069435    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e98548b_7fbe_45aa_a928_3ca569f02bb9.slice/crio-9739f7a62c06d6f3dd95dbcf92a5b613341fed36e654583ca6c12dca154c251c WatchSource:0}: Error finding container 9739f7a62c06d6f3dd95dbcf92a5b613341fed36e654583ca6c12dca154c251c: Status 404 returned error can't find the container with id 9739f7a62c06d6f3dd95dbcf92a5b613341fed36e654583ca6c12dca154c251c
Jul 24 13:20:15.067676 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:15.067608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv" event={"ID":"0e98548b-7fbe-45aa-a928-3ca569f02bb9","Type":"ContainerStarted","Data":"9739f7a62c06d6f3dd95dbcf92a5b613341fed36e654583ca6c12dca154c251c"}
Jul 24 13:20:16.074297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:16.074265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv" event={"ID":"0e98548b-7fbe-45aa-a928-3ca569f02bb9","Type":"ContainerStarted","Data":"1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd"}
Jul 24 13:20:16.074717 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:16.074333    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:16.075809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:16.075781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7" event={"ID":"4b1a7464-a447-4757-92bf-021df6eb99bc","Type":"ContainerStarted","Data":"3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213"}
Jul 24 13:20:16.075914 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:16.075901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:16.097381 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:16.097322    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv" podStartSLOduration=1.197053852 podStartE2EDuration="3.097300359s" totalImagesPullingTime="1.900246507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:20:16.092828837 +0000 UTC m=+1640.082147531" watchObservedRunningTime="2026-07-24 13:20:16.097300359 +0000 UTC m=+1640.086619056"
Jul 24 13:20:16.111873 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:16.111822    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7" podStartSLOduration=2.39693552 podStartE2EDuration="4.111805813s" totalImagesPullingTime="1.714870293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:20:16.111102523 +0000 UTC m=+1640.100421240" watchObservedRunningTime="2026-07-24 13:20:16.111805813 +0000 UTC m=+1640.101124551"
Jul 24 13:20:22.086464 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:22.086437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:27.082651 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:27.082622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:28.031383 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:28.031354    2579 patch_prober.go:28] interesting pod/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:36550->10.133.0.54:15021: read: connection reset by peer" start-of-body=
Jul 24 13:20:28.031536 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:28.031422    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:36550->10.133.0.54:15021: read: connection reset by peer"
Jul 24 13:20:28.148889 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:28.148851    2579 generic.go:356] "Generic (PLEG): container finished" podID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerID="f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e" exitCode=0
Jul 24 13:20:28.149277 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:28.148901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" event={"ID":"f47e7d9d-5203-4ea2-8f3f-a22abad357e0","Type":"ContainerDied","Data":"f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e"}
Jul 24 13:20:28.149342 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:28.149328    2579 scope.go:117] "RemoveContainer" containerID="f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e"
Jul 24 13:20:28.150437 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:28.150411    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:28.150500 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:28.150481    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:28.150536 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:28.150520    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:29.161767 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:29.161729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" event={"ID":"f47e7d9d-5203-4ea2-8f3f-a22abad357e0","Type":"ContainerStarted","Data":"aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f"}
Jul 24 13:20:29.164278 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:29.164259    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:29.164442 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:29.164423    2579 patch_prober.go:28] interesting pod/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 24 13:20:29.164482 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:29.164465    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 24 13:20:30.164467 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:30.164427    2579 patch_prober.go:28] interesting pod/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 24 13:20:30.164955 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:30.164500    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 24 13:20:31.164085 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:31.164051    2579 patch_prober.go:28] interesting pod/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 24 13:20:31.164299 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:31.164115    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 24 13:20:32.164355 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:32.164325    2579 patch_prober.go:28] interesting pod/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 24 13:20:32.164717 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:32.164381    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 24 13:20:33.167954 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:33.167911    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:33.168323 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:33.168199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:33.168943 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:33.168912    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:48.800886 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:48.800853    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"]
Jul 24 13:20:48.801392 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:48.801093    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7" podUID="4b1a7464-a447-4757-92bf-021df6eb99bc" containerName="maas-api" containerID="cri-o://3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213" gracePeriod=30
Jul 24 13:20:48.876910 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:48.876878    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"]
Jul 24 13:20:48.877126 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:48.877100    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv" podUID="0e98548b-7fbe-45aa-a928-3ca569f02bb9" containerName="payload-processing" containerID="cri-o://1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd" gracePeriod=30
Jul 24 13:20:49.131297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.131270    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:49.156906 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.156882    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:49.199312 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.199279    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kdlg\" (UniqueName: \"kubernetes.io/projected/0e98548b-7fbe-45aa-a928-3ca569f02bb9-kube-api-access-4kdlg\") pod \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\" (UID: \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\") "
Jul 24 13:20:49.199486 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.199319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls\") pod \"4b1a7464-a447-4757-92bf-021df6eb99bc\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") "
Jul 24 13:20:49.199486 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.199346    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e98548b-7fbe-45aa-a928-3ca569f02bb9-plugins-config-volume\") pod \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\" (UID: \"0e98548b-7fbe-45aa-a928-3ca569f02bb9\") "
Jul 24 13:20:49.199486 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.199420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1a7464-a447-4757-92bf-021df6eb99bc-openshift-service-ca\") pod \"4b1a7464-a447-4757-92bf-021df6eb99bc\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") "
Jul 24 13:20:49.199662 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.199493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gctjb\" (UniqueName: \"kubernetes.io/projected/4b1a7464-a447-4757-92bf-021df6eb99bc-kube-api-access-gctjb\") pod \"4b1a7464-a447-4757-92bf-021df6eb99bc\" (UID: \"4b1a7464-a447-4757-92bf-021df6eb99bc\") "
Jul 24 13:20:49.199761 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.199741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e98548b-7fbe-45aa-a928-3ca569f02bb9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e98548b-7fbe-45aa-a928-3ca569f02bb9" (UID: "0e98548b-7fbe-45aa-a928-3ca569f02bb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:20:49.199820 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.199795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1a7464-a447-4757-92bf-021df6eb99bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b1a7464-a447-4757-92bf-021df6eb99bc" (UID: "4b1a7464-a447-4757-92bf-021df6eb99bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:20:49.199868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.199818    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e98548b-7fbe-45aa-a928-3ca569f02bb9-plugins-config-volume\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:49.201623 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.201600    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e98548b-7fbe-45aa-a928-3ca569f02bb9-kube-api-access-4kdlg" (OuterVolumeSpecName: "kube-api-access-4kdlg") pod "0e98548b-7fbe-45aa-a928-3ca569f02bb9" (UID: "0e98548b-7fbe-45aa-a928-3ca569f02bb9"). InnerVolumeSpecName "kube-api-access-4kdlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:20:49.201720 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.201699    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1a7464-a447-4757-92bf-021df6eb99bc-kube-api-access-gctjb" (OuterVolumeSpecName: "kube-api-access-gctjb") pod "4b1a7464-a447-4757-92bf-021df6eb99bc" (UID: "4b1a7464-a447-4757-92bf-021df6eb99bc"). InnerVolumeSpecName "kube-api-access-gctjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:20:49.201913 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.201891    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b1a7464-a447-4757-92bf-021df6eb99bc" (UID: "4b1a7464-a447-4757-92bf-021df6eb99bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:20:49.268962 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.268913    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b1a7464-a447-4757-92bf-021df6eb99bc" containerID="3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213" exitCode=0
Jul 24 13:20:49.269133 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.268982    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"
Jul 24 13:20:49.269133 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.269024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7" event={"ID":"4b1a7464-a447-4757-92bf-021df6eb99bc","Type":"ContainerDied","Data":"3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213"}
Jul 24 13:20:49.269133 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.269082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7" event={"ID":"4b1a7464-a447-4757-92bf-021df6eb99bc","Type":"ContainerDied","Data":"8b5ee14aadf6b4e1c45bf2ab74ce74d32798ae9508fdae09b4d076dde543fe78"}
Jul 24 13:20:49.269133 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.269111    2579 scope.go:117] "RemoveContainer" containerID="3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213"
Jul 24 13:20:49.270400 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.270373    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e98548b-7fbe-45aa-a928-3ca569f02bb9" containerID="1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd" exitCode=0
Jul 24 13:20:49.270400 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.270405    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"
Jul 24 13:20:49.270557 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.270477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv" event={"ID":"0e98548b-7fbe-45aa-a928-3ca569f02bb9","Type":"ContainerDied","Data":"1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd"}
Jul 24 13:20:49.270557 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.270519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv" event={"ID":"0e98548b-7fbe-45aa-a928-3ca569f02bb9","Type":"ContainerDied","Data":"9739f7a62c06d6f3dd95dbcf92a5b613341fed36e654583ca6c12dca154c251c"}
Jul 24 13:20:49.279154 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.279134    2579 scope.go:117] "RemoveContainer" containerID="3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213"
Jul 24 13:20:49.279460 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:20:49.279434    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213\": container with ID starting with 3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213 not found: ID does not exist" containerID="3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213"
Jul 24 13:20:49.279544 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.279472    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213"} err="failed to get container status \"3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213\": rpc error: code = NotFound desc = could not find container \"3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213\": container with ID starting with 3f2b9151910881ba8652a3df4204852aa9af4a63b7b71b4d6c5caaa0511e0213 not found: ID does not exist"
Jul 24 13:20:49.279544 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.279494    2579 scope.go:117] "RemoveContainer" containerID="1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd"
Jul 24 13:20:49.287817 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.287798    2579 scope.go:117] "RemoveContainer" containerID="1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd"
Jul 24 13:20:49.288088 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:20:49.288070    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd\": container with ID starting with 1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd not found: ID does not exist" containerID="1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd"
Jul 24 13:20:49.288149 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.288094    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd"} err="failed to get container status \"1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd\": rpc error: code = NotFound desc = could not find container \"1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd\": container with ID starting with 1db0203e3025169ea1fec45f8341286f6bcb119abad28f7c26844239b02318dd not found: ID does not exist"
Jul 24 13:20:49.295702 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.295680    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"]
Jul 24 13:20:49.300146 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.300121    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acfcca72-56d8b488cf-656t7"]
Jul 24 13:20:49.300576 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.300544    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdlg\" (UniqueName: \"kubernetes.io/projected/0e98548b-7fbe-45aa-a928-3ca569f02bb9-kube-api-access-4kdlg\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:49.300576 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.300573    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1a7464-a447-4757-92bf-021df6eb99bc-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:49.300711 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.300589    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1a7464-a447-4757-92bf-021df6eb99bc-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:49.300711 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.300601    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctjb\" (UniqueName: \"kubernetes.io/projected/4b1a7464-a447-4757-92bf-021df6eb99bc-kube-api-access-gctjb\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:49.312577 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.312552    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"]
Jul 24 13:20:49.317523 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:49.317499    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acfcca72-78d4fc567b-m24pv"]
Jul 24 13:20:50.563352 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:50.563318    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e98548b-7fbe-45aa-a928-3ca569f02bb9" path="/var/lib/kubelet/pods/0e98548b-7fbe-45aa-a928-3ca569f02bb9/volumes"
Jul 24 13:20:50.563716 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:50.563685    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1a7464-a447-4757-92bf-021df6eb99bc" path="/var/lib/kubelet/pods/4b1a7464-a447-4757-92bf-021df6eb99bc/volumes"
Jul 24 13:20:53.381304 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:53.381273    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"]
Jul 24 13:20:53.381688 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:53.381465    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy" containerID="cri-o://aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f" gracePeriod=30
Jul 24 13:20:54.437456 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.437423    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"]
Jul 24 13:20:54.438003 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.437986    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1a7464-a447-4757-92bf-021df6eb99bc" containerName="maas-api"
Jul 24 13:20:54.438095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.438006    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1a7464-a447-4757-92bf-021df6eb99bc" containerName="maas-api"
Jul 24 13:20:54.438095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.438019    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e98548b-7fbe-45aa-a928-3ca569f02bb9" containerName="payload-processing"
Jul 24 13:20:54.438095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.438028    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e98548b-7fbe-45aa-a928-3ca569f02bb9" containerName="payload-processing"
Jul 24 13:20:54.438241 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.438112    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1a7464-a447-4757-92bf-021df6eb99bc" containerName="maas-api"
Jul 24 13:20:54.438241 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.438124    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e98548b-7fbe-45aa-a928-3ca569f02bb9" containerName="payload-processing"
Jul 24 13:20:54.442625 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.442604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.445290 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.445274    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-90d0d94f-gw-openshift-default-dockercfg-6b7xc\""
Jul 24 13:20:54.454958 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.454935    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"]
Jul 24 13:20:54.538734 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.538706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdn9\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-kube-api-access-5xdn9\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.538907 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.538739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c305a3-dce3-4082-99af-35049ac6a698-istio-podinfo\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.538907 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.538777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c305a3-dce3-4082-99af-35049ac6a698-istiod-ca-cert\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.538907 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.538813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-envoy\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.538907 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.538850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-credential-socket\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.539075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.538940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-data\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.539075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.538990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-socket\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.539075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.539005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-certs\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.539075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.539020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-istio-token\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.639437 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-envoy\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.639594 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-credential-socket\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.639594 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-data\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.639594 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-socket\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.639594 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-certs\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.639826 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-istio-token\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.639826 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdn9\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-kube-api-access-5xdn9\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.639826 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c305a3-dce3-4082-99af-35049ac6a698-istio-podinfo\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.640020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c305a3-dce3-4082-99af-35049ac6a698-istiod-ca-cert\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.640020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-certs\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.640020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.639903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-data\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.640136 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.640057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-socket\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.640387 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.640361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-credential-socket\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.640514 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.640499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c305a3-dce3-4082-99af-35049ac6a698-istiod-ca-cert\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.641867 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.641852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-envoy\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.642410 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.642392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c305a3-dce3-4082-99af-35049ac6a698-istio-podinfo\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.651524 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.651500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-istio-token\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.651956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.651916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdn9\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-kube-api-access-5xdn9\") pod \"e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") " pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.754051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.753976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:54.897290 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.897262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"]
Jul 24 13:20:54.898798 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:20:54.898774    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c305a3_dce3_4082_99af_35049ac6a698.slice/crio-870234f43f309faadaf6774cbb378ab387729d7954269145d413b3a4e4e2c748 WatchSource:0}: Error finding container 870234f43f309faadaf6774cbb378ab387729d7954269145d413b3a4e4e2c748: Status 404 returned error can't find the container with id 870234f43f309faadaf6774cbb378ab387729d7954269145d413b3a4e4e2c748
Jul 24 13:20:54.900800 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.900768    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:54.900941 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.900845    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:54.900941 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:54.900890    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:20:55.307727 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:55.307680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" event={"ID":"76c305a3-dce3-4082-99af-35049ac6a698","Type":"ContainerStarted","Data":"41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92"}
Jul 24 13:20:55.307933 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:55.307740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" event={"ID":"76c305a3-dce3-4082-99af-35049ac6a698","Type":"ContainerStarted","Data":"870234f43f309faadaf6774cbb378ab387729d7954269145d413b3a4e4e2c748"}
Jul 24 13:20:55.332555 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:55.332503    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" podStartSLOduration=1.33248973 podStartE2EDuration="1.33248973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:20:55.330850254 +0000 UTC m=+1679.320168950" watchObservedRunningTime="2026-07-24 13:20:55.33248973 +0000 UTC m=+1679.321808426"
Jul 24 13:20:55.754893 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:55.754864    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:55.759871 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:55.759844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:56.313072 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.313035    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:56.313868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.313852    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:20:56.979430 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.979393    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"]
Jul 24 13:20:56.984783 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.984761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:56.987515 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.987492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:20:56.987637 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.987527    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-90d0d94f-serving-cert\""
Jul 24 13:20:56.988473 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.988454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:20:56.988599 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.988509    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:20:56.993396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:56.993371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"]
Jul 24 13:20:57.061077 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.061039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/492c7d2a-a0ee-40e9-a09d-63e6080f104e-maas-api-tls\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.061246 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.061139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/492c7d2a-a0ee-40e9-a09d-63e6080f104e-openshift-service-ca\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.061246 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.061174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p99k\" (UniqueName: \"kubernetes.io/projected/492c7d2a-a0ee-40e9-a09d-63e6080f104e-kube-api-access-5p99k\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.161626 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.161591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/492c7d2a-a0ee-40e9-a09d-63e6080f104e-maas-api-tls\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.161772 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.161654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/492c7d2a-a0ee-40e9-a09d-63e6080f104e-openshift-service-ca\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.161772 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.161679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p99k\" (UniqueName: \"kubernetes.io/projected/492c7d2a-a0ee-40e9-a09d-63e6080f104e-kube-api-access-5p99k\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.162262 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.162240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/492c7d2a-a0ee-40e9-a09d-63e6080f104e-openshift-service-ca\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.164199 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.164182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/492c7d2a-a0ee-40e9-a09d-63e6080f104e-maas-api-tls\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.170401 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.170367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p99k\" (UniqueName: \"kubernetes.io/projected/492c7d2a-a0ee-40e9-a09d-63e6080f104e-kube-api-access-5p99k\") pod \"maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.297634 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.297552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:57.437768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:57.437742    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"]
Jul 24 13:20:57.439065 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:20:57.439038    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492c7d2a_a0ee_40e9_a09d_63e6080f104e.slice/crio-52d150c46905fe5bbf135fc1efdb4d2edc3aa59d2125e1043e85a2267dfd4671 WatchSource:0}: Error finding container 52d150c46905fe5bbf135fc1efdb4d2edc3aa59d2125e1043e85a2267dfd4671: Status 404 returned error can't find the container with id 52d150c46905fe5bbf135fc1efdb4d2edc3aa59d2125e1043e85a2267dfd4671
Jul 24 13:20:58.326291 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.326252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq" event={"ID":"492c7d2a-a0ee-40e9-a09d-63e6080f104e","Type":"ContainerStarted","Data":"41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63"}
Jul 24 13:20:58.326653 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.326335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:20:58.326653 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.326370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq" event={"ID":"492c7d2a-a0ee-40e9-a09d-63e6080f104e","Type":"ContainerStarted","Data":"52d150c46905fe5bbf135fc1efdb4d2edc3aa59d2125e1043e85a2267dfd4671"}
Jul 24 13:20:58.346758 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.346716    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq" podStartSLOduration=1.8935739360000001 podStartE2EDuration="2.34670434s" totalImagesPullingTime="453.130404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:20:58.345560679 +0000 UTC m=+1682.334879375" watchObservedRunningTime="2026-07-24 13:20:58.34670434 +0000 UTC m=+1682.336023035"
Jul 24 13:20:58.657389 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.657362    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:58.676297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istiod-ca-cert\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676483 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-podinfo\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676483 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx4w8\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-kube-api-access-jx4w8\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676483 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676412    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-token\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676483 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676437    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-credential-socket\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676483 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676469    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-data\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-envoy\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-certs\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-socket\") pod \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\" (UID: \"f47e7d9d-5203-4ea2-8f3f-a22abad357e0\") "
Jul 24 13:20:58.676741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:20:58.676970 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:20:58.677044 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676978    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:58.677044 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.676998    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:58.677044 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.677017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:20:58.677238 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.677140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:20:58.678167 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.678140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:20:58.678834 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.678794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-kube-api-access-jx4w8" (OuterVolumeSpecName: "kube-api-access-jx4w8") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "kube-api-access-jx4w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:20:58.679001 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.678874    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:20:58.679001 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.678988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:20:58.679297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.679273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f47e7d9d-5203-4ea2-8f3f-a22abad357e0" (UID: "f47e7d9d-5203-4ea2-8f3f-a22abad357e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:20:58.777671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.777644    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:58.777671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.777674    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx4w8\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-kube-api-access-jx4w8\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:58.777858 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.777684    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:58.777858 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.777692    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:58.777858 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.777700    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:58.777858 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.777708    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:58.777858 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:58.777716    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47e7d9d-5203-4ea2-8f3f-a22abad357e0-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:20:59.332669 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.332636    2579 generic.go:356] "Generic (PLEG): container finished" podID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerID="aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f" exitCode=0
Jul 24 13:20:59.333035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.332678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"
Jul 24 13:20:59.333035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.332724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" event={"ID":"f47e7d9d-5203-4ea2-8f3f-a22abad357e0","Type":"ContainerDied","Data":"aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f"}
Jul 24 13:20:59.333035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.332776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62" event={"ID":"f47e7d9d-5203-4ea2-8f3f-a22abad357e0","Type":"ContainerDied","Data":"85344ea138425c5e1da4faf0859bea5ed2c1fd0fe7f01b7074ecf2e94efa1f6f"}
Jul 24 13:20:59.333035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.332799    2579 scope.go:117] "RemoveContainer" containerID="aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f"
Jul 24 13:20:59.342264 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.342248    2579 scope.go:117] "RemoveContainer" containerID="f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e"
Jul 24 13:20:59.350641 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.350622    2579 scope.go:117] "RemoveContainer" containerID="aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f"
Jul 24 13:20:59.350873 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:20:59.350853    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f\": container with ID starting with aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f not found: ID does not exist" containerID="aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f"
Jul 24 13:20:59.350964 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.350885    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f"} err="failed to get container status \"aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f\": rpc error: code = NotFound desc = could not find container \"aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f\": container with ID starting with aa15b20a892efe503c8616105fe52ab65f0841748a2f616eabe5e02b3a74530f not found: ID does not exist"
Jul 24 13:20:59.350964 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.350907    2579 scope.go:117] "RemoveContainer" containerID="f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e"
Jul 24 13:20:59.351219 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:20:59.351189    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e\": container with ID starting with f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e not found: ID does not exist" containerID="f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e"
Jul 24 13:20:59.351331 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.351222    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e"} err="failed to get container status \"f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e\": rpc error: code = NotFound desc = could not find container \"f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e\": container with ID starting with f4d4b05d0a535cd99dd1cd7223f957f33be8abeb05cca49b1df03cae68830b3e not found: ID does not exist"
Jul 24 13:20:59.358728 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.358600    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"]
Jul 24 13:20:59.364678 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:20:59.364659    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-acfcca72-openshift-default-58d7cdfcb8-gst62"]
Jul 24 13:21:00.563057 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:00.563015    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" path="/var/lib/kubelet/pods/f47e7d9d-5203-4ea2-8f3f-a22abad357e0/volumes"
Jul 24 13:21:04.338349 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:04.338317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:21:11.314813 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:11.314779    2579 patch_prober.go:28] interesting pod/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 24 13:21:11.315249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:11.314849    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 24 13:21:11.400564 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:11.400492    2579 generic.go:356] "Generic (PLEG): container finished" podID="76c305a3-dce3-4082-99af-35049ac6a698" containerID="41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92" exitCode=0
Jul 24 13:21:11.400564 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:11.400520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" event={"ID":"76c305a3-dce3-4082-99af-35049ac6a698","Type":"ContainerDied","Data":"41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92"}
Jul 24 13:21:11.400983 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:11.400967    2579 scope.go:117] "RemoveContainer" containerID="41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92"
Jul 24 13:21:11.402111 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:11.402086    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:11.402170 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:11.402155    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:11.402219 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:11.402187    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:12.412462 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:12.411656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" event={"ID":"76c305a3-dce3-4082-99af-35049ac6a698","Type":"ContainerStarted","Data":"91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373"}
Jul 24 13:21:12.754728 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:12.754644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:21:12.754869 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:12.754837    2579 patch_prober.go:28] interesting pod/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 24 13:21:12.754915 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:12.754888    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 24 13:21:13.755161 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:13.755133    2579 patch_prober.go:28] interesting pod/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 24 13:21:13.755508 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:13.755183    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 24 13:21:14.755215 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:14.755184    2579 patch_prober.go:28] interesting pod/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 24 13:21:14.755558 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:14.755244    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 24 13:21:15.754466 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:15.754430    2579 patch_prober.go:28] interesting pod/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 24 13:21:15.754650 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:15.754479    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 24 13:21:16.757940 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:16.757894    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:21:16.758444 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:16.758425    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:21:16.759097 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:16.759078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:21:32.450877 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.450844    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"]
Jul 24 13:21:32.451269 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.451078    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq" podUID="492c7d2a-a0ee-40e9-a09d-63e6080f104e" containerName="maas-api" containerID="cri-o://41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63" gracePeriod=30
Jul 24 13:21:32.699463 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.699438    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:21:32.800469 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.800386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/492c7d2a-a0ee-40e9-a09d-63e6080f104e-openshift-service-ca\") pod \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") "
Jul 24 13:21:32.800469 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.800455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p99k\" (UniqueName: \"kubernetes.io/projected/492c7d2a-a0ee-40e9-a09d-63e6080f104e-kube-api-access-5p99k\") pod \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") "
Jul 24 13:21:32.800646 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.800566    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/492c7d2a-a0ee-40e9-a09d-63e6080f104e-maas-api-tls\") pod \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\" (UID: \"492c7d2a-a0ee-40e9-a09d-63e6080f104e\") "
Jul 24 13:21:32.800750 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.800725    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492c7d2a-a0ee-40e9-a09d-63e6080f104e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "492c7d2a-a0ee-40e9-a09d-63e6080f104e" (UID: "492c7d2a-a0ee-40e9-a09d-63e6080f104e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:21:32.800862 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.800842    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/492c7d2a-a0ee-40e9-a09d-63e6080f104e-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:32.802625 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.802596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492c7d2a-a0ee-40e9-a09d-63e6080f104e-kube-api-access-5p99k" (OuterVolumeSpecName: "kube-api-access-5p99k") pod "492c7d2a-a0ee-40e9-a09d-63e6080f104e" (UID: "492c7d2a-a0ee-40e9-a09d-63e6080f104e"). InnerVolumeSpecName "kube-api-access-5p99k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:21:32.802715 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.802657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492c7d2a-a0ee-40e9-a09d-63e6080f104e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "492c7d2a-a0ee-40e9-a09d-63e6080f104e" (UID: "492c7d2a-a0ee-40e9-a09d-63e6080f104e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:21:32.902228 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.902192    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p99k\" (UniqueName: \"kubernetes.io/projected/492c7d2a-a0ee-40e9-a09d-63e6080f104e-kube-api-access-5p99k\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:32.902228 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:32.902225    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/492c7d2a-a0ee-40e9-a09d-63e6080f104e-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:33.529409 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.529377    2579 generic.go:356] "Generic (PLEG): container finished" podID="492c7d2a-a0ee-40e9-a09d-63e6080f104e" containerID="41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63" exitCode=0
Jul 24 13:21:33.529802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.529421    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"
Jul 24 13:21:33.529802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.529465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq" event={"ID":"492c7d2a-a0ee-40e9-a09d-63e6080f104e","Type":"ContainerDied","Data":"41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63"}
Jul 24 13:21:33.529802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.529521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq" event={"ID":"492c7d2a-a0ee-40e9-a09d-63e6080f104e","Type":"ContainerDied","Data":"52d150c46905fe5bbf135fc1efdb4d2edc3aa59d2125e1043e85a2267dfd4671"}
Jul 24 13:21:33.529802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.529549    2579 scope.go:117] "RemoveContainer" containerID="41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63"
Jul 24 13:21:33.539441 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.539426    2579 scope.go:117] "RemoveContainer" containerID="41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63"
Jul 24 13:21:33.539674 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:21:33.539657    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63\": container with ID starting with 41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63 not found: ID does not exist" containerID="41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63"
Jul 24 13:21:33.539714 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.539683    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63"} err="failed to get container status \"41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63\": rpc error: code = NotFound desc = could not find container \"41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63\": container with ID starting with 41482e030cb09a72737d9bfdb4918a14e2a681637bdd6c3eb55b78ca1c758e63 not found: ID does not exist"
Jul 24 13:21:33.553632 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.553609    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"]
Jul 24 13:21:33.560016 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:33.558854    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-90d0d94f-79bf4f4848-pvsbq"]
Jul 24 13:21:34.562361 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:34.562324    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492c7d2a-a0ee-40e9-a09d-63e6080f104e" path="/var/lib/kubelet/pods/492c7d2a-a0ee-40e9-a09d-63e6080f104e/volumes"
Jul 24 13:21:37.102740 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.102706    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"]
Jul 24 13:21:37.103266 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.102936    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy" containerID="cri-o://91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373" gracePeriod=30
Jul 24 13:21:37.885956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.885901    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"]
Jul 24 13:21:37.886322 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.886307    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy"
Jul 24 13:21:37.886376 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.886325    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy"
Jul 24 13:21:37.886376 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.886352    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492c7d2a-a0ee-40e9-a09d-63e6080f104e" containerName="maas-api"
Jul 24 13:21:37.886376 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.886358    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="492c7d2a-a0ee-40e9-a09d-63e6080f104e" containerName="maas-api"
Jul 24 13:21:37.886474 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.886413    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="492c7d2a-a0ee-40e9-a09d-63e6080f104e" containerName="maas-api"
Jul 24 13:21:37.886474 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.886422    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy"
Jul 24 13:21:37.890778 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.890757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.893667 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.893641    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a54eb296-openshift-default-dockercfg-w6bb5\""
Jul 24 13:21:37.902841 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.902818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"]
Jul 24 13:21:37.947571 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-token\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.947763 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-envoy\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.947763 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-data\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.947763 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgdj5\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-kube-api-access-bgdj5\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.947763 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istiod-ca-cert\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.947763 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-socket\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.948058 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-podinfo\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.948058 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-credential-socket\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:37.948058 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:37.947880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-certs\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049126 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-envoy\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-data\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgdj5\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-kube-api-access-bgdj5\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istiod-ca-cert\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-socket\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-podinfo\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-credential-socket\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-certs\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-token\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049822 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-socket\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.049912 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-certs\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.050084 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.049979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-data\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.050540 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.050492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-credential-socket\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.050944 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.050886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istiod-ca-cert\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.052945 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.052904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-envoy\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.053085 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.053065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-podinfo\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.059673 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.059647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-token\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.060143 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.060117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgdj5\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-kube-api-access-bgdj5\") pod \"e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") " pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.204145 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.204108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:38.353939 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.353896    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"]
Jul 24 13:21:38.355990 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:21:38.355961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8a9367_0745_4b9c_8111_2b95fc1a0215.slice/crio-40138ff202cca99da070657e3353389e1b2640bb9cf27fc7bd0eaca87b0db6f3 WatchSource:0}: Error finding container 40138ff202cca99da070657e3353389e1b2640bb9cf27fc7bd0eaca87b0db6f3: Status 404 returned error can't find the container with id 40138ff202cca99da070657e3353389e1b2640bb9cf27fc7bd0eaca87b0db6f3
Jul 24 13:21:38.358169 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.358139    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:38.358268 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.358222    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:38.358330 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.358267    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:38.568157 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.568077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" event={"ID":"ae8a9367-0745-4b9c-8111-2b95fc1a0215","Type":"ContainerStarted","Data":"74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274"}
Jul 24 13:21:38.568157 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.568133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" event={"ID":"ae8a9367-0745-4b9c-8111-2b95fc1a0215","Type":"ContainerStarted","Data":"40138ff202cca99da070657e3353389e1b2640bb9cf27fc7bd0eaca87b0db6f3"}
Jul 24 13:21:38.586842 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:38.586767    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" podStartSLOduration=1.586750364 podStartE2EDuration="1.586750364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:21:38.58584777 +0000 UTC m=+1722.575166464" watchObservedRunningTime="2026-07-24 13:21:38.586750364 +0000 UTC m=+1722.576069059"
Jul 24 13:21:39.204307 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:39.204271    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:39.209366 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:39.209342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:39.566996 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:39.566898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:39.567971 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:39.567945    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:40.321333 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.321299    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"]
Jul 24 13:21:40.321701 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.321688    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy"
Jul 24 13:21:40.321701 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.321701    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy"
Jul 24 13:21:40.321774 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.321768    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47e7d9d-5203-4ea2-8f3f-a22abad357e0" containerName="istio-proxy"
Jul 24 13:21:40.324878 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.324858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.328725 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.328701    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:21:40.329969 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.329951    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:21:40.329969 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.329962    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:21:40.330111 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.329958    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a54eb296-serving-cert\""
Jul 24 13:21:40.347417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.347384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"]
Jul 24 13:21:40.369222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.369189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-openshift-service-ca\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.369398 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.369230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.369398 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.369257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrmz\" (UniqueName: \"kubernetes.io/projected/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-kube-api-access-mwrmz\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.470201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.470160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-openshift-service-ca\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.470201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.470200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.470434 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.470219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrmz\" (UniqueName: \"kubernetes.io/projected/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-kube-api-access-mwrmz\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.470434 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:21:40.470318    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-serving-cert: secret "maas-api-e2e-ait-a54eb296-serving-cert" not found
Jul 24 13:21:40.470434 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:21:40.470384    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls podName:fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66 nodeName:}" failed. No retries permitted until 2026-07-24 13:21:40.97036792 +0000 UTC m=+1724.959686594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls") pod "maas-api-e2e-ait-a54eb296-d55dd6d99-g926j" (UID: "fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66") : secret "maas-api-e2e-ait-a54eb296-serving-cert" not found
Jul 24 13:21:40.470814 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.470795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-openshift-service-ca\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.486760 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.486733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrmz\" (UniqueName: \"kubernetes.io/projected/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-kube-api-access-mwrmz\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.974913 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.974878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:40.977428 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:40.977408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls\") pod \"maas-api-e2e-ait-a54eb296-d55dd6d99-g926j\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:41.235448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:41.235353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:41.388251 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:41.388219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"]
Jul 24 13:21:41.390024 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:21:41.389989    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe11d9e8_a396_4aaf_b5ab_cf6faedc3d66.slice/crio-247596c00e96387a0b0729a09c55615ea1a3fd8d9bd3c943efef5a89897670ec WatchSource:0}: Error finding container 247596c00e96387a0b0729a09c55615ea1a3fd8d9bd3c943efef5a89897670ec: Status 404 returned error can't find the container with id 247596c00e96387a0b0729a09c55615ea1a3fd8d9bd3c943efef5a89897670ec
Jul 24 13:21:41.578348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:41.578262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j" event={"ID":"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66","Type":"ContainerStarted","Data":"247596c00e96387a0b0729a09c55615ea1a3fd8d9bd3c943efef5a89897670ec"}
Jul 24 13:21:42.354003 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.353981    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:21:42.387098 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387006    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c305a3-dce3-4082-99af-35049ac6a698-istio-podinfo\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387098 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387043    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-certs\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387098 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387064    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xdn9\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-kube-api-access-5xdn9\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387098 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387081    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-istio-token\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-socket\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387171    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-credential-socket\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387243    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c305a3-dce3-4082-99af-35049ac6a698-istiod-ca-cert\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-envoy\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:21:42.387419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-data\") pod \"76c305a3-dce3-4082-99af-35049ac6a698\" (UID: \"76c305a3-dce3-4082-99af-35049ac6a698\") "
Jul 24 13:21:42.387757 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:21:42.387757 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387749    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.387862 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.387824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:21:42.388222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.388189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c305a3-dce3-4082-99af-35049ac6a698-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:21:42.388687 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.388649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-data" (OuterVolumeSpecName: "istio-data") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:21:42.390387 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.390363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-istio-token" (OuterVolumeSpecName: "istio-token") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:21:42.390387 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.390370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c305a3-dce3-4082-99af-35049ac6a698-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:21:42.390548 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.390432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:21:42.390786 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.390761    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-kube-api-access-5xdn9" (OuterVolumeSpecName: "kube-api-access-5xdn9") pod "76c305a3-dce3-4082-99af-35049ac6a698" (UID: "76c305a3-dce3-4082-99af-35049ac6a698"). InnerVolumeSpecName "kube-api-access-5xdn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:21:42.488757 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.488725    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.488757 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.488754    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c305a3-dce3-4082-99af-35049ac6a698-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.488757 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.488765    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.489017 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.488772    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.489017 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.488780    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c305a3-dce3-4082-99af-35049ac6a698-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.489017 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.488789    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xdn9\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-kube-api-access-5xdn9\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.489017 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.488800    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c305a3-dce3-4082-99af-35049ac6a698-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.489017 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.488813    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c305a3-dce3-4082-99af-35049ac6a698-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:21:42.584529 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.584496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j" event={"ID":"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66","Type":"ContainerStarted","Data":"17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9"}
Jul 24 13:21:42.584684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.584560    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:42.586186 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.586163    2579 generic.go:356] "Generic (PLEG): container finished" podID="76c305a3-dce3-4082-99af-35049ac6a698" containerID="91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373" exitCode=0
Jul 24 13:21:42.586305 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.586202    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"
Jul 24 13:21:42.586305 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.586240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" event={"ID":"76c305a3-dce3-4082-99af-35049ac6a698","Type":"ContainerDied","Data":"91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373"}
Jul 24 13:21:42.586305 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.586290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp" event={"ID":"76c305a3-dce3-4082-99af-35049ac6a698","Type":"ContainerDied","Data":"870234f43f309faadaf6774cbb378ab387729d7954269145d413b3a4e4e2c748"}
Jul 24 13:21:42.586430 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.586319    2579 scope.go:117] "RemoveContainer" containerID="91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373"
Jul 24 13:21:42.596174 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.596155    2579 scope.go:117] "RemoveContainer" containerID="41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92"
Jul 24 13:21:42.604593 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.604577    2579 scope.go:117] "RemoveContainer" containerID="91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373"
Jul 24 13:21:42.604837 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:21:42.604817    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373\": container with ID starting with 91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373 not found: ID does not exist" containerID="91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373"
Jul 24 13:21:42.604909 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.604851    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373"} err="failed to get container status \"91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373\": rpc error: code = NotFound desc = could not find container \"91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373\": container with ID starting with 91d63a6dedd20ab7abb9d12244df64feba558f21796c124c889fba9e073d6373 not found: ID does not exist"
Jul 24 13:21:42.604909 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.604876    2579 scope.go:117] "RemoveContainer" containerID="41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92"
Jul 24 13:21:42.605135 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:21:42.605119    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92\": container with ID starting with 41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92 not found: ID does not exist" containerID="41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92"
Jul 24 13:21:42.605173 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.605141    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92"} err="failed to get container status \"41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92\": rpc error: code = NotFound desc = could not find container \"41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92\": container with ID starting with 41cb60a44828560e4f38ef8fe239a37cfab654a028896097a6f8110ed764cd92 not found: ID does not exist"
Jul 24 13:21:42.609970 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.609932    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j" podStartSLOduration=2.195737732 podStartE2EDuration="2.60990747s" totalImagesPullingTime="414.169738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:21:42.607806594 +0000 UTC m=+1726.597125289" watchObservedRunningTime="2026-07-24 13:21:42.60990747 +0000 UTC m=+1726.599226166"
Jul 24 13:21:42.623247 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.623225    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"]
Jul 24 13:21:42.627877 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:42.627856    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-90d0d94f-gw-openshift-default-cf799bf4d-szqjp"]
Jul 24 13:21:44.563272 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:44.563241    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c305a3-dce3-4082-99af-35049ac6a698" path="/var/lib/kubelet/pods/76c305a3-dce3-4082-99af-35049ac6a698/volumes"
Jul 24 13:21:48.609237 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:48.609200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:21:55.200613 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:55.200566    2579 patch_prober.go:28] interesting pod/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:44176->10.133.0.59:15021: read: connection reset by peer" start-of-body=
Jul 24 13:21:55.201016 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:55.200630    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:44176->10.133.0.59:15021: read: connection reset by peer"
Jul 24 13:21:55.660323 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:55.660290    2579 generic.go:356] "Generic (PLEG): container finished" podID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerID="74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274" exitCode=0
Jul 24 13:21:55.660509 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:55.660370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" event={"ID":"ae8a9367-0745-4b9c-8111-2b95fc1a0215","Type":"ContainerDied","Data":"74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274"}
Jul 24 13:21:55.660801 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:55.660785    2579 scope.go:117] "RemoveContainer" containerID="74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274"
Jul 24 13:21:55.661795 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:55.661770    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:55.661852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:55.661836    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:55.661894 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:55.661866    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:21:56.667111 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:56.667067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" event={"ID":"ae8a9367-0745-4b9c-8111-2b95fc1a0215","Type":"ContainerStarted","Data":"bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6"}
Jul 24 13:21:57.204759 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:57.204712    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:21:57.205123 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:57.204793    2579 patch_prober.go:28] interesting pod/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 24 13:21:57.205332 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:57.205290    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 24 13:21:58.205163 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:58.205125    2579 patch_prober.go:28] interesting pod/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 24 13:21:58.205750 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:58.205716    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 24 13:21:59.205395 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:59.205355    2579 patch_prober.go:28] interesting pod/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 24 13:21:59.205990 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:21:59.205953    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 24 13:22:00.205437 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:00.205399    2579 patch_prober.go:28] interesting pod/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 24 13:22:00.206026 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:00.205990    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 24 13:22:01.208612 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:01.208584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:22:01.209012 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:01.208837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:22:01.209591 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:01.209574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:22:16.493401 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.493369    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"]
Jul 24 13:22:16.493951 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.493570    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j" podUID="fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" containerName="maas-api" containerID="cri-o://17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9" gracePeriod=30
Jul 24 13:22:16.734131 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.734108    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:22:16.788896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.787555    2579 generic.go:356] "Generic (PLEG): container finished" podID="fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" containerID="17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9" exitCode=0
Jul 24 13:22:16.788896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.787718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j" event={"ID":"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66","Type":"ContainerDied","Data":"17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9"}
Jul 24 13:22:16.788896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.787805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j" event={"ID":"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66","Type":"ContainerDied","Data":"247596c00e96387a0b0729a09c55615ea1a3fd8d9bd3c943efef5a89897670ec"}
Jul 24 13:22:16.788896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.787838    2579 scope.go:117] "RemoveContainer" containerID="17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9"
Jul 24 13:22:16.788896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.788370    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"
Jul 24 13:22:16.798215 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.798188    2579 scope.go:117] "RemoveContainer" containerID="17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9"
Jul 24 13:22:16.798461 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:22:16.798438    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9\": container with ID starting with 17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9 not found: ID does not exist" containerID="17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9"
Jul 24 13:22:16.798461 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.798468    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9"} err="failed to get container status \"17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9\": rpc error: code = NotFound desc = could not find container \"17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9\": container with ID starting with 17199877f5d38110d02047f5f7292cb7ffd7f1c5975fdd6c3118b0a685406ff9 not found: ID does not exist"
Jul 24 13:22:16.894812 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.894779    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-openshift-service-ca\") pod \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") "
Jul 24 13:22:16.895000 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.894874    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls\") pod \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") "
Jul 24 13:22:16.895000 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.894943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwrmz\" (UniqueName: \"kubernetes.io/projected/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-kube-api-access-mwrmz\") pod \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\" (UID: \"fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66\") "
Jul 24 13:22:16.895245 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.895219    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" (UID: "fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:22:16.897081 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.897051    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" (UID: "fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:22:16.897200 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.897178    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-kube-api-access-mwrmz" (OuterVolumeSpecName: "kube-api-access-mwrmz") pod "fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" (UID: "fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66"). InnerVolumeSpecName "kube-api-access-mwrmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:22:16.995964 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.995931    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:16.995964 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.995957    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrmz\" (UniqueName: \"kubernetes.io/projected/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-kube-api-access-mwrmz\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:16.995964 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:16.995967    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:17.121547 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:17.121517    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"]
Jul 24 13:22:17.125954 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:17.125908    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a54eb296-d55dd6d99-g926j"]
Jul 24 13:22:18.563548 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:18.563513    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" path="/var/lib/kubelet/pods/fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66/volumes"
Jul 24 13:22:22.200062 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.200022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"]
Jul 24 13:22:22.200529 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.200284    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy" containerID="cri-o://bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6" gracePeriod=30
Jul 24 13:22:22.995260 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.995216    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"]
Jul 24 13:22:22.995668 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.995655    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy"
Jul 24 13:22:22.995711 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.995670    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy"
Jul 24 13:22:22.995711 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.995708    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" containerName="maas-api"
Jul 24 13:22:22.995780 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.995714    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" containerName="maas-api"
Jul 24 13:22:22.995780 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.995770    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy"
Jul 24 13:22:22.995843 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:22.995784    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe11d9e8-a396-4aaf-b5ab-cf6faedc3d66" containerName="maas-api"
Jul 24 13:22:23.000243 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.000224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.003889 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.003871    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-40670996-openshift-default-dockercfg-h5572\""
Jul 24 13:22:23.011981 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.011953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"]
Jul 24 13:22:23.149841 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.149804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-data\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.149841 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.149845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-istio-token\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.150152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.149878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-envoy\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.150152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.149975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342115e1-db08-41f4-9cce-cc6f148b0675-istiod-ca-cert\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.150152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.150019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342115e1-db08-41f4-9cce-cc6f148b0675-istio-podinfo\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.150152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.150105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hpc\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-kube-api-access-f2hpc\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.150152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.150138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-credential-socket\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.150366 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.150172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-socket\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.150366 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.150207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-certs\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.251688 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.251590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-data\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.251688 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.251638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-istio-token\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.251774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-envoy\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.251824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342115e1-db08-41f4-9cce-cc6f148b0675-istiod-ca-cert\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.251863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342115e1-db08-41f4-9cce-cc6f148b0675-istio-podinfo\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.251976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hpc\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-kube-api-access-f2hpc\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.252021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-credential-socket\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.252057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-data\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.252063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-socket\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.252176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-certs\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252566 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.252329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-credential-socket\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252566 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.252359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-socket\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252566 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.252527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-certs\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.252766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.252747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342115e1-db08-41f4-9cce-cc6f148b0675-istiod-ca-cert\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.254187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.254161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-envoy\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.254401 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.254385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342115e1-db08-41f4-9cce-cc6f148b0675-istio-podinfo\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.261444 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.261412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hpc\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-kube-api-access-f2hpc\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.261543 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.261500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-istio-token\") pod \"e2e-derive-40670996-openshift-default-b856c687b-t4mvh\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") " pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.312651 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.312619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:23.487622 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.487594    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"]
Jul 24 13:22:23.489338 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:22:23.489309    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342115e1_db08_41f4_9cce_cc6f148b0675.slice/crio-f980deb8ba879e86eba76246a67c969222437a190b7acf033e1cc0f02bbb556c WatchSource:0}: Error finding container f980deb8ba879e86eba76246a67c969222437a190b7acf033e1cc0f02bbb556c: Status 404 returned error can't find the container with id f980deb8ba879e86eba76246a67c969222437a190b7acf033e1cc0f02bbb556c
Jul 24 13:22:23.491621 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.491590    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:22:23.491717 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.491656    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:22:23.491717 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.491704    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:22:23.831207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.831113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" event={"ID":"342115e1-db08-41f4-9cce-cc6f148b0675","Type":"ContainerStarted","Data":"ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07"}
Jul 24 13:22:23.831207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:23.831181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" event={"ID":"342115e1-db08-41f4-9cce-cc6f148b0675","Type":"ContainerStarted","Data":"f980deb8ba879e86eba76246a67c969222437a190b7acf033e1cc0f02bbb556c"}
Jul 24 13:22:24.313420 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:24.313386    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:24.318310 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:24.318284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:24.347916 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:24.347873    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" podStartSLOduration=2.3478582599999998 podStartE2EDuration="2.34785826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:22:23.867084545 +0000 UTC m=+1767.856403241" watchObservedRunningTime="2026-07-24 13:22:24.34785826 +0000 UTC m=+1768.337176956"
Jul 24 13:22:24.835877 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:24.835844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:24.836645 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:24.836626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:25.451482 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.451453    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"]
Jul 24 13:22:25.451842 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.451812    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy"
Jul 24 13:22:25.451842 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.451823    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy"
Jul 24 13:22:25.451918 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.451889    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c305a3-dce3-4082-99af-35049ac6a698" containerName="istio-proxy"
Jul 24 13:22:25.454975 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.454958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.460206 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.460185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:22:25.460206 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.460195    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:22:25.473684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.473658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-40670996-serving-cert\""
Jul 24 13:22:25.474634 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.474609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:22:25.477336 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.477319    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"]
Jul 24 13:22:25.572491 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.572461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45772b92-29e7-49e6-bf20-f72fa1252750-maas-api-tls\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.572668 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.572525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45772b92-29e7-49e6-bf20-f72fa1252750-openshift-service-ca\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.572668 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.572550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftb6m\" (UniqueName: \"kubernetes.io/projected/45772b92-29e7-49e6-bf20-f72fa1252750-kube-api-access-ftb6m\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.673762 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.673728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45772b92-29e7-49e6-bf20-f72fa1252750-maas-api-tls\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.673971 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.673793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45772b92-29e7-49e6-bf20-f72fa1252750-openshift-service-ca\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.673971 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.673815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftb6m\" (UniqueName: \"kubernetes.io/projected/45772b92-29e7-49e6-bf20-f72fa1252750-kube-api-access-ftb6m\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.674506 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.674481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45772b92-29e7-49e6-bf20-f72fa1252750-openshift-service-ca\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.676283 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.676257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45772b92-29e7-49e6-bf20-f72fa1252750-maas-api-tls\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.691187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.691158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftb6m\" (UniqueName: \"kubernetes.io/projected/45772b92-29e7-49e6-bf20-f72fa1252750-kube-api-access-ftb6m\") pod \"maas-api-e2e-derive-40670996-78d48dbbd5-65psz\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.765276 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.765189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:25.909696 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:25.909669    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"]
Jul 24 13:22:25.911206 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:22:25.911185    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45772b92_29e7_49e6_bf20_f72fa1252750.slice/crio-b9bb186efa45fc35a515303704e43a99333a1804550992dd78c068325db8c5ce WatchSource:0}: Error finding container b9bb186efa45fc35a515303704e43a99333a1804550992dd78c068325db8c5ce: Status 404 returned error can't find the container with id b9bb186efa45fc35a515303704e43a99333a1804550992dd78c068325db8c5ce
Jul 24 13:22:26.848597 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:26.848538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz" event={"ID":"45772b92-29e7-49e6-bf20-f72fa1252750","Type":"ContainerStarted","Data":"2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10"}
Jul 24 13:22:26.849091 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:26.848615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz" event={"ID":"45772b92-29e7-49e6-bf20-f72fa1252750","Type":"ContainerStarted","Data":"b9bb186efa45fc35a515303704e43a99333a1804550992dd78c068325db8c5ce"}
Jul 24 13:22:26.849091 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:26.848957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:26.871037 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:26.870993    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz" podStartSLOduration=1.45543866 podStartE2EDuration="1.870979093s" totalImagesPullingTime="415.540433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:22:26.869530056 +0000 UTC m=+1770.858848752" watchObservedRunningTime="2026-07-24 13:22:26.870979093 +0000 UTC m=+1770.860297804"
Jul 24 13:22:27.463893 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.463837    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:22:27.592615 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-envoy\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.592804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-token\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.592804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-socket\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.592804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592703    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-podinfo\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.592804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgdj5\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-kube-api-access-bgdj5\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.592804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592782    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-certs\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.593099 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592825    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istiod-ca-cert\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.593099 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-data\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.593099 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592988    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-credential-socket\") pod \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\" (UID: \"ae8a9367-0745-4b9c-8111-2b95fc1a0215\") "
Jul 24 13:22:27.593099 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.592993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:22:27.593310 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.593265    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:22:27.593310 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.593276    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.593596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.593550    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:22:27.593664 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.593636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:22:27.594378 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.594340    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:22:27.594984 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.594958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:22:27.595209 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.595184    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:22:27.595475 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.595452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:22:27.595789 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.595768    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-kube-api-access-bgdj5" (OuterVolumeSpecName: "kube-api-access-bgdj5") pod "ae8a9367-0745-4b9c-8111-2b95fc1a0215" (UID: "ae8a9367-0745-4b9c-8111-2b95fc1a0215"). InnerVolumeSpecName "kube-api-access-bgdj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:22:27.694407 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.694373    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.694407 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.694411    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.694606 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.694426    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.694606 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.694441    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.694606 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.694452    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.694606 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.694465    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgdj5\" (UniqueName: \"kubernetes.io/projected/ae8a9367-0745-4b9c-8111-2b95fc1a0215-kube-api-access-bgdj5\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.694606 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.694477    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8a9367-0745-4b9c-8111-2b95fc1a0215-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.694606 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.694489    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8a9367-0745-4b9c-8111-2b95fc1a0215-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:22:27.855567 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.855471    2579 generic.go:356] "Generic (PLEG): container finished" podID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerID="bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6" exitCode=0
Jul 24 13:22:27.855567 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.855522    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"
Jul 24 13:22:27.856093 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.855560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" event={"ID":"ae8a9367-0745-4b9c-8111-2b95fc1a0215","Type":"ContainerDied","Data":"bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6"}
Jul 24 13:22:27.856093 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.855618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq" event={"ID":"ae8a9367-0745-4b9c-8111-2b95fc1a0215","Type":"ContainerDied","Data":"40138ff202cca99da070657e3353389e1b2640bb9cf27fc7bd0eaca87b0db6f3"}
Jul 24 13:22:27.856093 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.855647    2579 scope.go:117] "RemoveContainer" containerID="bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6"
Jul 24 13:22:27.865200 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.865185    2579 scope.go:117] "RemoveContainer" containerID="74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274"
Jul 24 13:22:27.873430 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.873407    2579 scope.go:117] "RemoveContainer" containerID="bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6"
Jul 24 13:22:27.873698 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:22:27.873678    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6\": container with ID starting with bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6 not found: ID does not exist" containerID="bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6"
Jul 24 13:22:27.873780 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.873706    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6"} err="failed to get container status \"bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6\": rpc error: code = NotFound desc = could not find container \"bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6\": container with ID starting with bd156fe7414603cb6c2419fb98e08ec9a4a6c393e6395c970fda12392ba06aa6 not found: ID does not exist"
Jul 24 13:22:27.873780 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.873724    2579 scope.go:117] "RemoveContainer" containerID="74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274"
Jul 24 13:22:27.873951 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:22:27.873917    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274\": container with ID starting with 74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274 not found: ID does not exist" containerID="74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274"
Jul 24 13:22:27.874005 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.873958    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274"} err="failed to get container status \"74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274\": rpc error: code = NotFound desc = could not find container \"74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274\": container with ID starting with 74d478fc42447e76073daa6d22bbc267f02b40bb16d1978281d2735ee91d3274 not found: ID does not exist"
Jul 24 13:22:27.879581 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.879560    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"]
Jul 24 13:22:27.886153 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:27.886134    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a54eb296-openshift-default-7d85d79ffb-zdzcq"]
Jul 24 13:22:28.564404 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:28.564366    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" path="/var/lib/kubelet/pods/ae8a9367-0745-4b9c-8111-2b95fc1a0215/volumes"
Jul 24 13:22:32.861007 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:32.860971    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:22:39.837370 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:39.837331    2579 patch_prober.go:28] interesting pod/e2e-derive-40670996-openshift-default-b856c687b-t4mvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 24 13:22:39.837800 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:39.837401    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 24 13:22:39.931372 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:39.931340    2579 generic.go:356] "Generic (PLEG): container finished" podID="342115e1-db08-41f4-9cce-cc6f148b0675" containerID="ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07" exitCode=0
Jul 24 13:22:39.931544 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:39.931424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" event={"ID":"342115e1-db08-41f4-9cce-cc6f148b0675","Type":"ContainerDied","Data":"ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07"}
Jul 24 13:22:39.931837 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:39.931823    2579 scope.go:117] "RemoveContainer" containerID="ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07"
Jul 24 13:22:39.933005 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:39.932979    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:22:39.933073 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:39.933062    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:22:39.933116 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:39.933101    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:22:40.942890 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:40.942851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" event={"ID":"342115e1-db08-41f4-9cce-cc6f148b0675","Type":"ContainerStarted","Data":"e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d"}
Jul 24 13:22:41.313314 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:41.313207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:41.313508 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:41.313474    2579 patch_prober.go:28] interesting pod/e2e-derive-40670996-openshift-default-b856c687b-t4mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 24 13:22:41.313586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:41.313536    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 24 13:22:42.313597 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:42.313548    2579 patch_prober.go:28] interesting pod/e2e-derive-40670996-openshift-default-b856c687b-t4mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 24 13:22:42.314126 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:42.313612    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 24 13:22:43.313439 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:43.313401    2579 patch_prober.go:28] interesting pod/e2e-derive-40670996-openshift-default-b856c687b-t4mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 24 13:22:43.313677 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:43.313460    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 24 13:22:44.314007 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:44.313968    2579 patch_prober.go:28] interesting pod/e2e-derive-40670996-openshift-default-b856c687b-t4mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 24 13:22:44.314397 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:44.314032    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 24 13:22:45.317678 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:45.317645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:45.318076 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:45.317915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:22:45.318621 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:22:45.318603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:23:00.685624 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:00.685589    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"]
Jul 24 13:23:00.686074 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:00.685795    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz" podUID="45772b92-29e7-49e6-bf20-f72fa1252750" containerName="maas-api" containerID="cri-o://2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10" gracePeriod=30
Jul 24 13:23:00.940575 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:00.940519    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:23:01.057310 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.057273    2579 generic.go:356] "Generic (PLEG): container finished" podID="45772b92-29e7-49e6-bf20-f72fa1252750" containerID="2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10" exitCode=0
Jul 24 13:23:01.057488 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.057322    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"
Jul 24 13:23:01.057488 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.057360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz" event={"ID":"45772b92-29e7-49e6-bf20-f72fa1252750","Type":"ContainerDied","Data":"2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10"}
Jul 24 13:23:01.057488 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.057417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz" event={"ID":"45772b92-29e7-49e6-bf20-f72fa1252750","Type":"ContainerDied","Data":"b9bb186efa45fc35a515303704e43a99333a1804550992dd78c068325db8c5ce"}
Jul 24 13:23:01.057488 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.057438    2579 scope.go:117] "RemoveContainer" containerID="2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10"
Jul 24 13:23:01.066438 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.066424    2579 scope.go:117] "RemoveContainer" containerID="2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10"
Jul 24 13:23:01.066689 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:23:01.066656    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10\": container with ID starting with 2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10 not found: ID does not exist" containerID="2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10"
Jul 24 13:23:01.066727 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.066697    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10"} err="failed to get container status \"2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10\": rpc error: code = NotFound desc = could not find container \"2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10\": container with ID starting with 2cf72c93890c8ab4e03d8a7bad92615831dc32f103fa37181b326a44d5f3bd10 not found: ID does not exist"
Jul 24 13:23:01.096746 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.096722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftb6m\" (UniqueName: \"kubernetes.io/projected/45772b92-29e7-49e6-bf20-f72fa1252750-kube-api-access-ftb6m\") pod \"45772b92-29e7-49e6-bf20-f72fa1252750\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") "
Jul 24 13:23:01.096916 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.096765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45772b92-29e7-49e6-bf20-f72fa1252750-openshift-service-ca\") pod \"45772b92-29e7-49e6-bf20-f72fa1252750\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") "
Jul 24 13:23:01.096916 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.096795    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45772b92-29e7-49e6-bf20-f72fa1252750-maas-api-tls\") pod \"45772b92-29e7-49e6-bf20-f72fa1252750\" (UID: \"45772b92-29e7-49e6-bf20-f72fa1252750\") "
Jul 24 13:23:01.097099 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.097072    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45772b92-29e7-49e6-bf20-f72fa1252750-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45772b92-29e7-49e6-bf20-f72fa1252750" (UID: "45772b92-29e7-49e6-bf20-f72fa1252750"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:23:01.097204 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.097167    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45772b92-29e7-49e6-bf20-f72fa1252750-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:01.099050 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.099021    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45772b92-29e7-49e6-bf20-f72fa1252750-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45772b92-29e7-49e6-bf20-f72fa1252750" (UID: "45772b92-29e7-49e6-bf20-f72fa1252750"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:23:01.099050 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.099034    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45772b92-29e7-49e6-bf20-f72fa1252750-kube-api-access-ftb6m" (OuterVolumeSpecName: "kube-api-access-ftb6m") pod "45772b92-29e7-49e6-bf20-f72fa1252750" (UID: "45772b92-29e7-49e6-bf20-f72fa1252750"). InnerVolumeSpecName "kube-api-access-ftb6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:23:01.197789 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.197709    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45772b92-29e7-49e6-bf20-f72fa1252750-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:01.197789 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.197736    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftb6m\" (UniqueName: \"kubernetes.io/projected/45772b92-29e7-49e6-bf20-f72fa1252750-kube-api-access-ftb6m\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:01.382343 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.382314    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"]
Jul 24 13:23:01.387763 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:01.387735    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-40670996-78d48dbbd5-65psz"]
Jul 24 13:23:02.564547 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:02.564516    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45772b92-29e7-49e6-bf20-f72fa1252750" path="/var/lib/kubelet/pods/45772b92-29e7-49e6-bf20-f72fa1252750/volumes"
Jul 24 13:23:05.446748 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:05.446710    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"]
Jul 24 13:23:05.447135 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:05.446992    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy" containerID="cri-o://e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d" gracePeriod=30
Jul 24 13:23:10.702899 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.702867    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:23:10.781428 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-certs\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.781428 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-data\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.781428 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-socket\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.781686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-credential-socket\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.781686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781475    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-envoy\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.781686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-istio-token\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.781686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781563    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342115e1-db08-41f4-9cce-cc6f148b0675-istio-podinfo\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.781686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342115e1-db08-41f4-9cce-cc6f148b0675-istiod-ca-cert\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.781686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2hpc\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-kube-api-access-f2hpc\") pod \"342115e1-db08-41f4-9cce-cc6f148b0675\" (UID: \"342115e1-db08-41f4-9cce-cc6f148b0675\") "
Jul 24 13:23:10.782028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:23:10.782028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:23:10.782028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.781765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:23:10.782028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.782013    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:10.782236 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.782035    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:10.782236 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.782053    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:10.782374 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.782349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342115e1-db08-41f4-9cce-cc6f148b0675-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:23:10.782562 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.782472    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-data" (OuterVolumeSpecName: "istio-data") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:23:10.784306 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.784259    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/342115e1-db08-41f4-9cce-cc6f148b0675-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:23:10.784426 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.784321    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-kube-api-access-f2hpc" (OuterVolumeSpecName: "kube-api-access-f2hpc") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "kube-api-access-f2hpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:23:10.784489 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.784457    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-istio-token" (OuterVolumeSpecName: "istio-token") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:23:10.784537 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.784516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "342115e1-db08-41f4-9cce-cc6f148b0675" (UID: "342115e1-db08-41f4-9cce-cc6f148b0675"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:23:10.882775 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.882746    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:10.882775 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.882769    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:10.882775 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.882778    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342115e1-db08-41f4-9cce-cc6f148b0675-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:10.882775 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.882786    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342115e1-db08-41f4-9cce-cc6f148b0675-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:10.883056 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.882794    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2hpc\" (UniqueName: \"kubernetes.io/projected/342115e1-db08-41f4-9cce-cc6f148b0675-kube-api-access-f2hpc\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:10.883056 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:10.882802    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342115e1-db08-41f4-9cce-cc6f148b0675-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:23:11.115649 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.115555    2579 generic.go:356] "Generic (PLEG): container finished" podID="342115e1-db08-41f4-9cce-cc6f148b0675" containerID="e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d" exitCode=0
Jul 24 13:23:11.115649 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.115596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" event={"ID":"342115e1-db08-41f4-9cce-cc6f148b0675","Type":"ContainerDied","Data":"e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d"}
Jul 24 13:23:11.115649 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.115637    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"
Jul 24 13:23:11.115649 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.115650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh" event={"ID":"342115e1-db08-41f4-9cce-cc6f148b0675","Type":"ContainerDied","Data":"f980deb8ba879e86eba76246a67c969222437a190b7acf033e1cc0f02bbb556c"}
Jul 24 13:23:11.115981 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.115678    2579 scope.go:117] "RemoveContainer" containerID="e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d"
Jul 24 13:23:11.124991 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.124971    2579 scope.go:117] "RemoveContainer" containerID="ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07"
Jul 24 13:23:11.133249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.133233    2579 scope.go:117] "RemoveContainer" containerID="e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d"
Jul 24 13:23:11.133505 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:23:11.133486    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d\": container with ID starting with e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d not found: ID does not exist" containerID="e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d"
Jul 24 13:23:11.133544 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.133515    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d"} err="failed to get container status \"e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d\": rpc error: code = NotFound desc = could not find container \"e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d\": container with ID starting with e5a31383d6339d75d844c7a43fc227a79ee819973972d31cc9a7d5f25e0bc82d not found: ID does not exist"
Jul 24 13:23:11.133544 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.133534    2579 scope.go:117] "RemoveContainer" containerID="ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07"
Jul 24 13:23:11.133772 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:23:11.133754    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07\": container with ID starting with ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07 not found: ID does not exist" containerID="ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07"
Jul 24 13:23:11.133825 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.133776    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07"} err="failed to get container status \"ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07\": rpc error: code = NotFound desc = could not find container \"ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07\": container with ID starting with ce807ff8cad620c41931402cded8213836fc1a4caf2e9af12c1b9f49a23f1a07 not found: ID does not exist"
Jul 24 13:23:11.160872 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.160843    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"]
Jul 24 13:23:11.165151 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:11.165131    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-40670996-openshift-default-b856c687b-t4mvh"]
Jul 24 13:23:12.562674 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:23:12.562640    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" path="/var/lib/kubelet/pods/342115e1-db08-41f4-9cce-cc6f148b0675/volumes"
Jul 24 13:24:56.082461 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082418    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"]
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082814    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy"
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082826    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy"
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082846    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy"
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082851    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy"
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082857    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45772b92-29e7-49e6-bf20-f72fa1252750" containerName="maas-api"
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082862    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45772b92-29e7-49e6-bf20-f72fa1252750" containerName="maas-api"
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082873    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy"
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082881    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy"
Jul 24 13:24:56.082960 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082959    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy"
Jul 24 13:24:56.083250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082970    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy"
Jul 24 13:24:56.083250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082979    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy"
Jul 24 13:24:56.083250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082986    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8a9367-0745-4b9c-8111-2b95fc1a0215" containerName="istio-proxy"
Jul 24 13:24:56.083250 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.082992    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45772b92-29e7-49e6-bf20-f72fa1252750" containerName="maas-api"
Jul 24 13:24:56.086224 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.086206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.088982 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.088960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b1c83c54-openshift-default-dockercfg-47ltb\""
Jul 24 13:24:56.114300 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.114266    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"]
Jul 24 13:24:56.199309 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-data\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.199481 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-credential-socket\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.199481 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-podinfo\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.199481 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-socket\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.199636 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-token\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.199636 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-envoy\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.199636 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99ed4c8-6872-40c1-b859-8b41babf1d72-istiod-ca-cert\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.199636 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-certs\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.199772 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.199644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltcht\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-kube-api-access-ltcht\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.300758 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.300722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-data\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.300969 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.300776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-credential-socket\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.300969 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.300892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-podinfo\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.300969 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.300952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-socket\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301127 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.300987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-token\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301127 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-envoy\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301127 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99ed4c8-6872-40c1-b859-8b41babf1d72-istiod-ca-cert\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301127 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-certs\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301335 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltcht\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-kube-api-access-ltcht\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301335 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-credential-socket\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301443 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-socket\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301540 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-certs\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301578 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-data\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.301747 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.301723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99ed4c8-6872-40c1-b859-8b41babf1d72-istiod-ca-cert\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.303530 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.303511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-envoy\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.303678 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.303661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-podinfo\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.312584 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.312561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-token\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.312749 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.312724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltcht\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-kube-api-access-ltcht\") pod \"e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") " pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.400122 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.400029    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b1c83c54-openshift-default-dockercfg-47ltb\""
Jul 24 13:24:56.407570 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.407543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:56.561426 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:24:56.561390    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99ed4c8_6872_40c1_b859_8b41babf1d72.slice/crio-2cbaa77c7c2d704893ca7491928a72dbeeb4f996dba6cbadcb87bd81be78f780 WatchSource:0}: Error finding container 2cbaa77c7c2d704893ca7491928a72dbeeb4f996dba6cbadcb87bd81be78f780: Status 404 returned error can't find the container with id 2cbaa77c7c2d704893ca7491928a72dbeeb4f996dba6cbadcb87bd81be78f780
Jul 24 13:24:56.563647 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.563603    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:24:56.563734 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.563709    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:24:56.563774 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.563750    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:24:56.565335 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.565316    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"]
Jul 24 13:24:56.682415 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.682368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" event={"ID":"f99ed4c8-6872-40c1-b859-8b41babf1d72","Type":"ContainerStarted","Data":"271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0"}
Jul 24 13:24:56.682625 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.682425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" event={"ID":"f99ed4c8-6872-40c1-b859-8b41babf1d72","Type":"ContainerStarted","Data":"2cbaa77c7c2d704893ca7491928a72dbeeb4f996dba6cbadcb87bd81be78f780"}
Jul 24 13:24:56.706353 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:56.706295    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" podStartSLOduration=0.706279417 podStartE2EDuration="706.279417ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:24:56.703659123 +0000 UTC m=+1920.692977821" watchObservedRunningTime="2026-07-24 13:24:56.706279417 +0000 UTC m=+1920.695598114"
Jul 24 13:24:57.408022 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:57.407980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:57.412801 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:57.412777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:57.688940 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:57.688837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:57.689867 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:57.689841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:24:58.956648 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.956610    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"]
Jul 24 13:24:58.957075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.957008    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy"
Jul 24 13:24:58.957075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.957021    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="342115e1-db08-41f4-9cce-cc6f148b0675" containerName="istio-proxy"
Jul 24 13:24:58.959759 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.959740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:58.962777 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.962650    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:24:58.962777 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.962703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:24:58.962995 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.962979    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b1c83c54-serving-cert\""
Jul 24 13:24:58.963699 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.963682    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:24:58.972087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:58.972062    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"]
Jul 24 13:24:59.028442 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.028400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.028442 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.028440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblwn\" (UniqueName: \"kubernetes.io/projected/10bf2b27-c055-478f-ab06-bb42f1243c9d-kube-api-access-cblwn\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.028685 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.028477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bf2b27-c055-478f-ab06-bb42f1243c9d-openshift-service-ca\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.129064 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.129019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.129064 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.129067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblwn\" (UniqueName: \"kubernetes.io/projected/10bf2b27-c055-478f-ab06-bb42f1243c9d-kube-api-access-cblwn\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.129332 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.129107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bf2b27-c055-478f-ab06-bb42f1243c9d-openshift-service-ca\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.129332 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:24:59.129182    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-serving-cert: secret "maas-api-e2e-mt-b1c83c54-serving-cert" not found
Jul 24 13:24:59.129332 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:24:59.129248    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls podName:10bf2b27-c055-478f-ab06-bb42f1243c9d nodeName:}" failed. No retries permitted until 2026-07-24 13:24:59.629231869 +0000 UTC m=+1923.618550548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls") pod "maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r" (UID: "10bf2b27-c055-478f-ab06-bb42f1243c9d") : secret "maas-api-e2e-mt-b1c83c54-serving-cert" not found
Jul 24 13:24:59.129818 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.129795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bf2b27-c055-478f-ab06-bb42f1243c9d-openshift-service-ca\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.141885 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.141854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblwn\" (UniqueName: \"kubernetes.io/projected/10bf2b27-c055-478f-ab06-bb42f1243c9d-kube-api-access-cblwn\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.633703 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.633664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.636387 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.636362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls\") pod \"maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:24:59.873048 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:24:59.873000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:25:00.031656 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:00.031627    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"]
Jul 24 13:25:00.033451 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:25:00.033418    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bf2b27_c055_478f_ab06_bb42f1243c9d.slice/crio-bab6df3651ae585d74a5511762332fc4d725e462c5f8e2f506c4c946f715eca9 WatchSource:0}: Error finding container bab6df3651ae585d74a5511762332fc4d725e462c5f8e2f506c4c946f715eca9: Status 404 returned error can't find the container with id bab6df3651ae585d74a5511762332fc4d725e462c5f8e2f506c4c946f715eca9
Jul 24 13:25:00.714287 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:00.714247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r" event={"ID":"10bf2b27-c055-478f-ab06-bb42f1243c9d","Type":"ContainerStarted","Data":"8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08"}
Jul 24 13:25:00.714449 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:00.714307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r" event={"ID":"10bf2b27-c055-478f-ab06-bb42f1243c9d","Type":"ContainerStarted","Data":"bab6df3651ae585d74a5511762332fc4d725e462c5f8e2f506c4c946f715eca9"}
Jul 24 13:25:00.714449 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:00.714333    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:25:00.733002 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:00.732879    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r" podStartSLOduration=2.297898284 podStartE2EDuration="2.732859749s" totalImagesPullingTime="434.961465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:25:00.732219946 +0000 UTC m=+1924.721538644" watchObservedRunningTime="2026-07-24 13:25:00.732859749 +0000 UTC m=+1924.722178448"
Jul 24 13:25:06.725808 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:06.725779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:25:09.763492 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:09.763459    2579 generic.go:356] "Generic (PLEG): container finished" podID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerID="271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0" exitCode=0
Jul 24 13:25:09.763899 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:09.763541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" event={"ID":"f99ed4c8-6872-40c1-b859-8b41babf1d72","Type":"ContainerDied","Data":"271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0"}
Jul 24 13:25:09.763963 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:09.763946    2579 scope.go:117] "RemoveContainer" containerID="271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0"
Jul 24 13:25:09.765010 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:09.764976    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:25:09.765075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:09.765051    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:25:09.765115 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:09.765086    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:25:10.770901 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:10.770867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" event={"ID":"f99ed4c8-6872-40c1-b859-8b41babf1d72","Type":"ContainerStarted","Data":"dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874"}
Jul 24 13:25:11.409222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:11.409153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:25:11.410809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:11.410779    2579 patch_prober.go:28] interesting pod/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 24 13:25:11.411051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:11.411023    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 24 13:25:12.408580 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:12.408088    2579 patch_prober.go:28] interesting pod/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 24 13:25:12.408580 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:12.408173    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 24 13:25:12.694177 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:12.694078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:25:13.408426 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:13.408381    2579 patch_prober.go:28] interesting pod/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 24 13:25:13.408636 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:13.408454    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 24 13:25:14.408732 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:14.408690    2579 patch_prober.go:28] interesting pod/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 24 13:25:14.409389 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:14.409353    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 24 13:25:15.411585 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:15.411556    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:25:15.412557 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:15.412531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:25:21.850358 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:21.850318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"]
Jul 24 13:25:21.850847 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:21.850566    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r" podUID="10bf2b27-c055-478f-ab06-bb42f1243c9d" containerName="maas-api" containerID="cri-o://8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08" gracePeriod=30
Jul 24 13:25:22.109726 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.109657    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:25:22.130264 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.130236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bf2b27-c055-478f-ab06-bb42f1243c9d-openshift-service-ca\") pod \"10bf2b27-c055-478f-ab06-bb42f1243c9d\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") "
Jul 24 13:25:22.130642 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.130586    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bf2b27-c055-478f-ab06-bb42f1243c9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10bf2b27-c055-478f-ab06-bb42f1243c9d" (UID: "10bf2b27-c055-478f-ab06-bb42f1243c9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:25:22.131003 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.130979    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls\") pod \"10bf2b27-c055-478f-ab06-bb42f1243c9d\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") "
Jul 24 13:25:22.131118 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.131026    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cblwn\" (UniqueName: \"kubernetes.io/projected/10bf2b27-c055-478f-ab06-bb42f1243c9d-kube-api-access-cblwn\") pod \"10bf2b27-c055-478f-ab06-bb42f1243c9d\" (UID: \"10bf2b27-c055-478f-ab06-bb42f1243c9d\") "
Jul 24 13:25:22.131378 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.131310    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bf2b27-c055-478f-ab06-bb42f1243c9d-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:22.133696 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.133663    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bf2b27-c055-478f-ab06-bb42f1243c9d-kube-api-access-cblwn" (OuterVolumeSpecName: "kube-api-access-cblwn") pod "10bf2b27-c055-478f-ab06-bb42f1243c9d" (UID: "10bf2b27-c055-478f-ab06-bb42f1243c9d"). InnerVolumeSpecName "kube-api-access-cblwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:25:22.134157 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.134128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10bf2b27-c055-478f-ab06-bb42f1243c9d" (UID: "10bf2b27-c055-478f-ab06-bb42f1243c9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:25:22.231816 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.231782    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblwn\" (UniqueName: \"kubernetes.io/projected/10bf2b27-c055-478f-ab06-bb42f1243c9d-kube-api-access-cblwn\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:22.231816 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.231812    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bf2b27-c055-478f-ab06-bb42f1243c9d-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:22.841343 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.841250    2579 generic.go:356] "Generic (PLEG): container finished" podID="10bf2b27-c055-478f-ab06-bb42f1243c9d" containerID="8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08" exitCode=0
Jul 24 13:25:22.841343 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.841308    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"
Jul 24 13:25:22.841343 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.841330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r" event={"ID":"10bf2b27-c055-478f-ab06-bb42f1243c9d","Type":"ContainerDied","Data":"8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08"}
Jul 24 13:25:22.841565 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.841406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r" event={"ID":"10bf2b27-c055-478f-ab06-bb42f1243c9d","Type":"ContainerDied","Data":"bab6df3651ae585d74a5511762332fc4d725e462c5f8e2f506c4c946f715eca9"}
Jul 24 13:25:22.841565 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.841428    2579 scope.go:117] "RemoveContainer" containerID="8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08"
Jul 24 13:25:22.850689 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.850660    2579 scope.go:117] "RemoveContainer" containerID="8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08"
Jul 24 13:25:22.851078 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:25:22.850971    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08\": container with ID starting with 8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08 not found: ID does not exist" containerID="8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08"
Jul 24 13:25:22.851078 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.851000    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08"} err="failed to get container status \"8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08\": rpc error: code = NotFound desc = could not find container \"8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08\": container with ID starting with 8e23b2c5b6c3e0aae8dcf85a7714f001c3cd57fb8fcd604717f142a7b1244e08 not found: ID does not exist"
Jul 24 13:25:22.863082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.863054    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"]
Jul 24 13:25:22.868246 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:22.868218    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c83c54-f767d58d5-d7s6r"]
Jul 24 13:25:24.562738 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:24.562699    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10bf2b27-c055-478f-ab06-bb42f1243c9d" path="/var/lib/kubelet/pods/10bf2b27-c055-478f-ab06-bb42f1243c9d/volumes"
Jul 24 13:25:33.580420 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:33.580385    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"]
Jul 24 13:25:33.581012 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:33.580604    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy" containerID="cri-o://dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874" gracePeriod=30
Jul 24 13:25:38.841818 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.841753    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:25:38.887822 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.887794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-data\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.887822 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.887827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltcht\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-kube-api-access-ltcht\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.888094 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.887852    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-podinfo\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.888094 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.887888    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-certs\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.888094 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-credential-socket\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.888094 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888108    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-token\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.888094 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99ed4c8-6872-40c1-b859-8b41babf1d72-istiod-ca-cert\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.888479 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-envoy\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.888479 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:25:38.888479 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888285    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-socket\") pod \"f99ed4c8-6872-40c1-b859-8b41babf1d72\" (UID: \"f99ed4c8-6872-40c1-b859-8b41babf1d72\") "
Jul 24 13:25:38.888479 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888447    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:25:38.888674 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888659    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:38.888730 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888681    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:38.888730 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:25:38.888895 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-data" (OuterVolumeSpecName: "istio-data") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:25:38.889100 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.888952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99ed4c8-6872-40c1-b859-8b41babf1d72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:25:38.890480 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.890449    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-token" (OuterVolumeSpecName: "istio-token") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:25:38.890588 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.890554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-kube-api-access-ltcht" (OuterVolumeSpecName: "kube-api-access-ltcht") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "kube-api-access-ltcht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:25:38.890852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.890831    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:25:38.890944 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.890837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f99ed4c8-6872-40c1-b859-8b41babf1d72" (UID: "f99ed4c8-6872-40c1-b859-8b41babf1d72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:25:38.929145 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.929100    2579 generic.go:356] "Generic (PLEG): container finished" podID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerID="dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874" exitCode=0
Jul 24 13:25:38.929305 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.929170    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"
Jul 24 13:25:38.929305 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.929190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" event={"ID":"f99ed4c8-6872-40c1-b859-8b41babf1d72","Type":"ContainerDied","Data":"dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874"}
Jul 24 13:25:38.929305 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.929259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989" event={"ID":"f99ed4c8-6872-40c1-b859-8b41babf1d72","Type":"ContainerDied","Data":"2cbaa77c7c2d704893ca7491928a72dbeeb4f996dba6cbadcb87bd81be78f780"}
Jul 24 13:25:38.929466 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.929289    2579 scope.go:117] "RemoveContainer" containerID="dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874"
Jul 24 13:25:38.939095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.939076    2579 scope.go:117] "RemoveContainer" containerID="271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0"
Jul 24 13:25:38.947619 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.947599    2579 scope.go:117] "RemoveContainer" containerID="dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874"
Jul 24 13:25:38.947883 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:25:38.947862    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874\": container with ID starting with dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874 not found: ID does not exist" containerID="dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874"
Jul 24 13:25:38.947951 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.947893    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874"} err="failed to get container status \"dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874\": rpc error: code = NotFound desc = could not find container \"dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874\": container with ID starting with dd339ddf81829f2f39d6f59ff87d5740027f68b064fe12af27a51c2d5ce7f874 not found: ID does not exist"
Jul 24 13:25:38.947951 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.947911    2579 scope.go:117] "RemoveContainer" containerID="271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0"
Jul 24 13:25:38.948211 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:25:38.948192    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0\": container with ID starting with 271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0 not found: ID does not exist" containerID="271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0"
Jul 24 13:25:38.948269 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.948221    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0"} err="failed to get container status \"271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0\": rpc error: code = NotFound desc = could not find container \"271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0\": container with ID starting with 271a001aa85ee83f69ab51d7d5c93f296b39bfe923ced9bc5d71ab3e7a4afba0 not found: ID does not exist"
Jul 24 13:25:38.954330 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.954308    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"]
Jul 24 13:25:38.958679 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.958651    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b1c83c54-openshift-default-7bf978cb56-vj989"]
Jul 24 13:25:38.989737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.989715    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:38.989737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.989736    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99ed4c8-6872-40c1-b859-8b41babf1d72-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:38.989870 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.989744    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:38.989870 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.989752    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:38.989870 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.989759    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:38.989870 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.989770    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltcht\" (UniqueName: \"kubernetes.io/projected/f99ed4c8-6872-40c1-b859-8b41babf1d72-kube-api-access-ltcht\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:38.989870 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:38.989778    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99ed4c8-6872-40c1-b859-8b41babf1d72-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:25:40.563206 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:40.563165    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" path="/var/lib/kubelet/pods/f99ed4c8-6872-40c1-b859-8b41babf1d72/volumes"
Jul 24 13:25:54.373627 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.373535    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"]
Jul 24 13:25:54.374058 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374039    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10bf2b27-c055-478f-ab06-bb42f1243c9d" containerName="maas-api"
Jul 24 13:25:54.374114 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374062    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="10bf2b27-c055-478f-ab06-bb42f1243c9d" containerName="maas-api"
Jul 24 13:25:54.374114 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374082    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy"
Jul 24 13:25:54.374114 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374092    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy"
Jul 24 13:25:54.374222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374121    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy"
Jul 24 13:25:54.374222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374130    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy"
Jul 24 13:25:54.374222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374204    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy"
Jul 24 13:25:54.374222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374213    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="10bf2b27-c055-478f-ab06-bb42f1243c9d" containerName="maas-api"
Jul 24 13:25:54.374222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.374219    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f99ed4c8-6872-40c1-b859-8b41babf1d72" containerName="istio-proxy"
Jul 24 13:25:54.377490 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.377475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.380262 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.380242    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-416b8ad5-openshift-default-dockercfg-5fhwr\""
Jul 24 13:25:54.389796 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.389770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"]
Jul 24 13:25:54.529177 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5btb\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-kube-api-access-v5btb\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.529177 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-podinfo\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.529459 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-credential-socket\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.529459 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-envoy\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.529459 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-socket\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.529459 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-data\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.529459 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e55defe-4edd-4434-ae00-d53d23f7bd55-istiod-ca-cert\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.529459 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-token\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.529678 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.529461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-certs\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630139 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-socket\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630139 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-data\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630139 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e55defe-4edd-4434-ae00-d53d23f7bd55-istiod-ca-cert\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630434 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-token\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630434 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-certs\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630434 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5btb\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-kube-api-access-v5btb\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630434 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-podinfo\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630434 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-data\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630676 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-credential-socket\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630676 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-envoy\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630676 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-socket\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630676 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-certs\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630884 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-credential-socket\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.630884 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.630825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e55defe-4edd-4434-ae00-d53d23f7bd55-istiod-ca-cert\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.632844 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.632823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-envoy\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.633082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.633062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-podinfo\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.640607 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.640584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5btb\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-kube-api-access-v5btb\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.644107 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.644086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-token\") pod \"e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") " pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.687253 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.687211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:54.851201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.851174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"]
Jul 24 13:25:54.852915 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:25:54.852889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e55defe_4edd_4434_ae00_d53d23f7bd55.slice/crio-d19bb3aa81befefdf4373b2117051b96cd338b3c344e80683783c6e7434f7718 WatchSource:0}: Error finding container d19bb3aa81befefdf4373b2117051b96cd338b3c344e80683783c6e7434f7718: Status 404 returned error can't find the container with id d19bb3aa81befefdf4373b2117051b96cd338b3c344e80683783c6e7434f7718
Jul 24 13:25:54.855357 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.855321    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:25:54.855440 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.855404    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:25:54.855484 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:54.855444    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:25:55.015833 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:55.015795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" event={"ID":"0e55defe-4edd-4434-ae00-d53d23f7bd55","Type":"ContainerStarted","Data":"fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842"}
Jul 24 13:25:55.016032 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:55.015854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" event={"ID":"0e55defe-4edd-4434-ae00-d53d23f7bd55","Type":"ContainerStarted","Data":"d19bb3aa81befefdf4373b2117051b96cd338b3c344e80683783c6e7434f7718"}
Jul 24 13:25:55.039591 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:55.039538    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" podStartSLOduration=1.039519395 podStartE2EDuration="1.039519395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:25:55.037260493 +0000 UTC m=+1979.026579189" watchObservedRunningTime="2026-07-24 13:25:55.039519395 +0000 UTC m=+1979.028838090"
Jul 24 13:25:55.687418 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:55.687376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:55.692118 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:55.692091    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:56.021351 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:56.021258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:56.022377 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:56.022352    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:25:57.068695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.068609    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"]
Jul 24 13:25:57.073686 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.073662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.076465 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.076444    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:25:57.076584 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.076488    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:25:57.077506 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.077487    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-416b8ad5-serving-cert\""
Jul 24 13:25:57.077613 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.077532    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:25:57.084125 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.084100    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"]
Jul 24 13:25:57.257561 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.257528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmnl9\" (UniqueName: \"kubernetes.io/projected/d50a3d62-964f-4221-9d46-8f18c08a763e-kube-api-access-vmnl9\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.257741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.257586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50a3d62-964f-4221-9d46-8f18c08a763e-maas-api-tls\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.257741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.257690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50a3d62-964f-4221-9d46-8f18c08a763e-openshift-service-ca\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.358673 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.358576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50a3d62-964f-4221-9d46-8f18c08a763e-openshift-service-ca\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.358673 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.358637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnl9\" (UniqueName: \"kubernetes.io/projected/d50a3d62-964f-4221-9d46-8f18c08a763e-kube-api-access-vmnl9\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.358673 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.358675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50a3d62-964f-4221-9d46-8f18c08a763e-maas-api-tls\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.359313 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.359289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50a3d62-964f-4221-9d46-8f18c08a763e-openshift-service-ca\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.361215 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.361193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50a3d62-964f-4221-9d46-8f18c08a763e-maas-api-tls\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.367531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.367505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnl9\" (UniqueName: \"kubernetes.io/projected/d50a3d62-964f-4221-9d46-8f18c08a763e-kube-api-access-vmnl9\") pod \"maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.385488 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.385468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:57.522684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.522651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"]
Jul 24 13:25:57.523747 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:25:57.523718    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50a3d62_964f_4221_9d46_8f18c08a763e.slice/crio-88eef60f601fb28cf5087b5b530927281d66eab619b4e27583343b4278c399e3 WatchSource:0}: Error finding container 88eef60f601fb28cf5087b5b530927281d66eab619b4e27583343b4278c399e3: Status 404 returned error can't find the container with id 88eef60f601fb28cf5087b5b530927281d66eab619b4e27583343b4278c399e3
Jul 24 13:25:57.525140 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:57.525125    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:25:58.036634 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:58.036525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc" event={"ID":"d50a3d62-964f-4221-9d46-8f18c08a763e","Type":"ContainerStarted","Data":"88eef60f601fb28cf5087b5b530927281d66eab619b4e27583343b4278c399e3"}
Jul 24 13:25:59.042706 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:59.042676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc" event={"ID":"d50a3d62-964f-4221-9d46-8f18c08a763e","Type":"ContainerStarted","Data":"d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd"}
Jul 24 13:25:59.043122 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:59.042789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:25:59.067444 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:25:59.067385    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc" podStartSLOduration=1.608124953 podStartE2EDuration="2.067367528s" totalImagesPullingTime="459.242575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:25:59.067052149 +0000 UTC m=+1983.056370849" watchObservedRunningTime="2026-07-24 13:25:59.067367528 +0000 UTC m=+1983.056686224"
Jul 24 13:26:05.029775 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.029734    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"]
Jul 24 13:26:05.033384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.033368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.037433 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.037406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-416b8ad5\"/\"default-dockercfg-jwcbq\""
Jul 24 13:26:05.037570 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.037406    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-416b8ad5\"/\"openshift-service-ca.crt\""
Jul 24 13:26:05.037570 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.037486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-416b8ad5\"/\"e2e-lifecycle-model-416b8ad5-kserve-self-signed-certs\""
Jul 24 13:26:05.037570 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.037490    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-416b8ad5\"/\"kube-root-ca.crt\""
Jul 24 13:26:05.051573 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.051550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"]
Jul 24 13:26:05.053703 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.053685    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:26:05.230102 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.230068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573dae6c-568d-4138-8bab-50cdcafbed86-tls-certs\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.230102 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.230109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-dshm\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.230333 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.230127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzdtn\" (UniqueName: \"kubernetes.io/projected/573dae6c-568d-4138-8bab-50cdcafbed86-kube-api-access-xzdtn\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.230333 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.230227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-model-cache\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.230333 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.230257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-tmp-dir\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.230434 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.230353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-home\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.330823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.330736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-home\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.330823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.330781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573dae6c-568d-4138-8bab-50cdcafbed86-tls-certs\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.331035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.330898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-dshm\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.331035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.330944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzdtn\" (UniqueName: \"kubernetes.io/projected/573dae6c-568d-4138-8bab-50cdcafbed86-kube-api-access-xzdtn\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.331035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.331008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-model-cache\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.331035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.331030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-tmp-dir\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.331222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.331198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-home\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.331388 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.331366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-model-cache\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.331469 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.331446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-tmp-dir\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.333251 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.333230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-dshm\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.333522 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.333505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573dae6c-568d-4138-8bab-50cdcafbed86-tls-certs\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.341384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.341360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzdtn\" (UniqueName: \"kubernetes.io/projected/573dae6c-568d-4138-8bab-50cdcafbed86-kube-api-access-xzdtn\") pod \"e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") " pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.343506 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.343489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:05.547419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:05.547395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"]
Jul 24 13:26:05.549152 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:26:05.549125    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod573dae6c_568d_4138_8bab_50cdcafbed86.slice/crio-3ddbba96336973494b71964122e01023b8fda1231ceb1d3627f20938d2e7d842 WatchSource:0}: Error finding container 3ddbba96336973494b71964122e01023b8fda1231ceb1d3627f20938d2e7d842: Status 404 returned error can't find the container with id 3ddbba96336973494b71964122e01023b8fda1231ceb1d3627f20938d2e7d842
Jul 24 13:26:06.079288 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:06.079250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6" event={"ID":"573dae6c-568d-4138-8bab-50cdcafbed86","Type":"ContainerStarted","Data":"aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056"}
Jul 24 13:26:06.079631 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:06.079294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6" event={"ID":"573dae6c-568d-4138-8bab-50cdcafbed86","Type":"ContainerStarted","Data":"3ddbba96336973494b71964122e01023b8fda1231ceb1d3627f20938d2e7d842"}
Jul 24 13:26:11.151523 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:11.151209    2579 patch_prober.go:28] interesting pod/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": read tcp 10.133.0.2:36848->10.133.0.65:15021: read: connection reset by peer" start-of-body=
Jul 24 13:26:11.151523 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:11.151287    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": read tcp 10.133.0.2:36848->10.133.0.65:15021: read: connection reset by peer"
Jul 24 13:26:12.119864 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:12.119827    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerID="fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842" exitCode=0
Jul 24 13:26:12.120078 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:12.119873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" event={"ID":"0e55defe-4edd-4434-ae00-d53d23f7bd55","Type":"ContainerDied","Data":"fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842"}
Jul 24 13:26:12.120405 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:12.120387    2579 scope.go:117] "RemoveContainer" containerID="fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842"
Jul 24 13:26:12.121487 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:12.121452    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:26:12.121592 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:12.121547    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:26:12.121645 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:12.121597    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:26:12.144369 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:12.144325    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6" podStartSLOduration=7.144311063 podStartE2EDuration="7.144311063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:26:06.105191922 +0000 UTC m=+1990.094510620" watchObservedRunningTime="2026-07-24 13:26:12.144311063 +0000 UTC m=+1996.133629775"
Jul 24 13:26:13.128102 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:13.128063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" event={"ID":"0e55defe-4edd-4434-ae00-d53d23f7bd55","Type":"ContainerStarted","Data":"1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb"}
Jul 24 13:26:13.687843 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:13.687804    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:26:13.688087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:13.688061    2579 patch_prober.go:28] interesting pod/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 24 13:26:13.688168 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:13.688121    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 24 13:26:14.687689 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:14.687646    2579 patch_prober.go:28] interesting pod/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 24 13:26:14.688200 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:14.687719    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 24 13:26:15.344271 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:15.344229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:15.344271 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:15.344280    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:15.356390 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:15.356367    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:15.688521 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:15.688480    2579 patch_prober.go:28] interesting pod/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 24 13:26:15.689010 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:15.688542    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 24 13:26:16.155569 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:16.155545    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:26:16.687704 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:16.687669    2579 patch_prober.go:28] interesting pod/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 24 13:26:16.687987 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:16.687952    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 24 13:26:17.691596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:17.691566    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:26:17.692063 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:17.692041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:26:17.692718 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:17.692704    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:26:41.198728 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.198679    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"]
Jul 24 13:26:41.199376 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.198960    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc" podUID="d50a3d62-964f-4221-9d46-8f18c08a763e" containerName="maas-api" containerID="cri-o://d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd" gracePeriod=30
Jul 24 13:26:41.445444 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.445335    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:26:41.576594 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.576522    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50a3d62-964f-4221-9d46-8f18c08a763e-maas-api-tls\") pod \"d50a3d62-964f-4221-9d46-8f18c08a763e\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") "
Jul 24 13:26:41.576735 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.576597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmnl9\" (UniqueName: \"kubernetes.io/projected/d50a3d62-964f-4221-9d46-8f18c08a763e-kube-api-access-vmnl9\") pod \"d50a3d62-964f-4221-9d46-8f18c08a763e\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") "
Jul 24 13:26:41.576735 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.576615    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50a3d62-964f-4221-9d46-8f18c08a763e-openshift-service-ca\") pod \"d50a3d62-964f-4221-9d46-8f18c08a763e\" (UID: \"d50a3d62-964f-4221-9d46-8f18c08a763e\") "
Jul 24 13:26:41.577020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.576983    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50a3d62-964f-4221-9d46-8f18c08a763e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d50a3d62-964f-4221-9d46-8f18c08a763e" (UID: "d50a3d62-964f-4221-9d46-8f18c08a763e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:26:41.578885 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.578862    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50a3d62-964f-4221-9d46-8f18c08a763e-kube-api-access-vmnl9" (OuterVolumeSpecName: "kube-api-access-vmnl9") pod "d50a3d62-964f-4221-9d46-8f18c08a763e" (UID: "d50a3d62-964f-4221-9d46-8f18c08a763e"). InnerVolumeSpecName "kube-api-access-vmnl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:26:41.578961 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.578865    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50a3d62-964f-4221-9d46-8f18c08a763e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d50a3d62-964f-4221-9d46-8f18c08a763e" (UID: "d50a3d62-964f-4221-9d46-8f18c08a763e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:26:41.677296 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.677264    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d50a3d62-964f-4221-9d46-8f18c08a763e-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:26:41.677296 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.677291    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmnl9\" (UniqueName: \"kubernetes.io/projected/d50a3d62-964f-4221-9d46-8f18c08a763e-kube-api-access-vmnl9\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:26:41.677296 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:41.677304    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50a3d62-964f-4221-9d46-8f18c08a763e-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:26:42.293249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.293215    2579 generic.go:356] "Generic (PLEG): container finished" podID="d50a3d62-964f-4221-9d46-8f18c08a763e" containerID="d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd" exitCode=0
Jul 24 13:26:42.293737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.293269    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"
Jul 24 13:26:42.293737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.293297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc" event={"ID":"d50a3d62-964f-4221-9d46-8f18c08a763e","Type":"ContainerDied","Data":"d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd"}
Jul 24 13:26:42.293737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.293368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc" event={"ID":"d50a3d62-964f-4221-9d46-8f18c08a763e","Type":"ContainerDied","Data":"88eef60f601fb28cf5087b5b530927281d66eab619b4e27583343b4278c399e3"}
Jul 24 13:26:42.293737 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.293403    2579 scope.go:117] "RemoveContainer" containerID="d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd"
Jul 24 13:26:42.302977 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.302961    2579 scope.go:117] "RemoveContainer" containerID="d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd"
Jul 24 13:26:42.303206 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:26:42.303185    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd\": container with ID starting with d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd not found: ID does not exist" containerID="d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd"
Jul 24 13:26:42.303272 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.303218    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd"} err="failed to get container status \"d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd\": rpc error: code = NotFound desc = could not find container \"d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd\": container with ID starting with d8640dad1191e2f1b9b7ff3a2e50a16b39328121d58d7c8797af6af51a8fecdd not found: ID does not exist"
Jul 24 13:26:42.317894 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.317868    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"]
Jul 24 13:26:42.323201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.323179    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-416b8ad5-bcbd7ddcf-qhtxc"]
Jul 24 13:26:42.563076 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:42.562993    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50a3d62-964f-4221-9d46-8f18c08a763e" path="/var/lib/kubelet/pods/d50a3d62-964f-4221-9d46-8f18c08a763e/volumes"
Jul 24 13:26:54.000733 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:26:54.000697    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"]
Jul 24 13:27:09.020084 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.020016    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6" podUID="573dae6c-568d-4138-8bab-50cdcafbed86" containerName="main" containerID="cri-o://aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056" gracePeriod=45
Jul 24 13:27:09.274706 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.274651    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:27:09.319000 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.318969    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-dshm\") pod \"573dae6c-568d-4138-8bab-50cdcafbed86\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") "
Jul 24 13:27:09.319149 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzdtn\" (UniqueName: \"kubernetes.io/projected/573dae6c-568d-4138-8bab-50cdcafbed86-kube-api-access-xzdtn\") pod \"573dae6c-568d-4138-8bab-50cdcafbed86\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") "
Jul 24 13:27:09.319149 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-home\") pod \"573dae6c-568d-4138-8bab-50cdcafbed86\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") "
Jul 24 13:27:09.319149 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319102    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-tmp-dir\") pod \"573dae6c-568d-4138-8bab-50cdcafbed86\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") "
Jul 24 13:27:09.319309 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-model-cache\") pod \"573dae6c-568d-4138-8bab-50cdcafbed86\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") "
Jul 24 13:27:09.319309 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573dae6c-568d-4138-8bab-50cdcafbed86-tls-certs\") pod \"573dae6c-568d-4138-8bab-50cdcafbed86\" (UID: \"573dae6c-568d-4138-8bab-50cdcafbed86\") "
Jul 24 13:27:09.319414 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319301    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "573dae6c-568d-4138-8bab-50cdcafbed86" (UID: "573dae6c-568d-4138-8bab-50cdcafbed86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:09.319414 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-home" (OuterVolumeSpecName: "home") pod "573dae6c-568d-4138-8bab-50cdcafbed86" (UID: "573dae6c-568d-4138-8bab-50cdcafbed86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:09.319572 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319526    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-model-cache" (OuterVolumeSpecName: "model-cache") pod "573dae6c-568d-4138-8bab-50cdcafbed86" (UID: "573dae6c-568d-4138-8bab-50cdcafbed86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:09.319764 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319720    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-home\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:09.319764 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319741    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-tmp-dir\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:09.319764 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.319750    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-model-cache\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:09.321380 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.321349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573dae6c-568d-4138-8bab-50cdcafbed86-kube-api-access-xzdtn" (OuterVolumeSpecName: "kube-api-access-xzdtn") pod "573dae6c-568d-4138-8bab-50cdcafbed86" (UID: "573dae6c-568d-4138-8bab-50cdcafbed86"). InnerVolumeSpecName "kube-api-access-xzdtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:27:09.321495 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.321380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-dshm" (OuterVolumeSpecName: "dshm") pod "573dae6c-568d-4138-8bab-50cdcafbed86" (UID: "573dae6c-568d-4138-8bab-50cdcafbed86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:09.321556 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.321528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573dae6c-568d-4138-8bab-50cdcafbed86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "573dae6c-568d-4138-8bab-50cdcafbed86" (UID: "573dae6c-568d-4138-8bab-50cdcafbed86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:27:09.420310 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.420278    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573dae6c-568d-4138-8bab-50cdcafbed86-dshm\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:09.420310 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.420304    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzdtn\" (UniqueName: \"kubernetes.io/projected/573dae6c-568d-4138-8bab-50cdcafbed86-kube-api-access-xzdtn\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:09.420310 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.420315    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573dae6c-568d-4138-8bab-50cdcafbed86-tls-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:09.439774 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.439743    2579 generic.go:356] "Generic (PLEG): container finished" podID="573dae6c-568d-4138-8bab-50cdcafbed86" containerID="aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056" exitCode=0
Jul 24 13:27:09.439912 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.439790    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"
Jul 24 13:27:09.439912 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.439830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6" event={"ID":"573dae6c-568d-4138-8bab-50cdcafbed86","Type":"ContainerDied","Data":"aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056"}
Jul 24 13:27:09.439912 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.439873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6" event={"ID":"573dae6c-568d-4138-8bab-50cdcafbed86","Type":"ContainerDied","Data":"3ddbba96336973494b71964122e01023b8fda1231ceb1d3627f20938d2e7d842"}
Jul 24 13:27:09.439912 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.439891    2579 scope.go:117] "RemoveContainer" containerID="aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056"
Jul 24 13:27:09.449475 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.449458    2579 scope.go:117] "RemoveContainer" containerID="aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056"
Jul 24 13:27:09.449710 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:27:09.449695    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056\": container with ID starting with aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056 not found: ID does not exist" containerID="aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056"
Jul 24 13:27:09.449748 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.449719    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056"} err="failed to get container status \"aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056\": rpc error: code = NotFound desc = could not find container \"aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056\": container with ID starting with aa2594d4bf158eca530c0b4817e5bf71f9cdded5aa22ea1853c6e191c2d6e056 not found: ID does not exist"
Jul 24 13:27:09.464026 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.463993    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"]
Jul 24 13:27:09.468858 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:09.468837    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-416b8ad5/e2e-lifecycle-model-416b8ad5-kserve-bd8c6dfc9-zjzz6"]
Jul 24 13:27:10.563869 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:10.563838    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="573dae6c-568d-4138-8bab-50cdcafbed86" path="/var/lib/kubelet/pods/573dae6c-568d-4138-8bab-50cdcafbed86/volumes"
Jul 24 13:27:39.166612 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:39.166577    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"]
Jul 24 13:27:39.167205 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:39.166784    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy" containerID="cri-o://1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb" gracePeriod=30
Jul 24 13:27:44.423956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.423912    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:27:44.522273 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5btb\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-kube-api-access-v5btb\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522273 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522225    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-certs\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522273 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522268    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-podinfo\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522291    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-socket\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522328    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-credential-socket\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-token\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-envoy\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e55defe-4edd-4434-ae00-d53d23f7bd55-istiod-ca-cert\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-data\") pod \"0e55defe-4edd-4434-ae00-d53d23f7bd55\" (UID: \"0e55defe-4edd-4434-ae00-d53d23f7bd55\") "
Jul 24 13:27:44.522531 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522473    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:44.522875 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522653    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:44.522875 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:44.522875 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522838    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.522875 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522857    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.522875 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522870    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.523130 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.522908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e55defe-4edd-4434-ae00-d53d23f7bd55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:27:44.523253 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.523232    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:44.524634 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.524601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-kube-api-access-v5btb" (OuterVolumeSpecName: "kube-api-access-v5btb") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "kube-api-access-v5btb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:27:44.525126 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.525094    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:27:44.525235 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.525176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:27:44.525235 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.525179    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e55defe-4edd-4434-ae00-d53d23f7bd55" (UID: "0e55defe-4edd-4434-ae00-d53d23f7bd55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:27:44.619888 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.619861    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerID="1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb" exitCode=0
Jul 24 13:27:44.620041 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.619911    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"
Jul 24 13:27:44.620041 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.619959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" event={"ID":"0e55defe-4edd-4434-ae00-d53d23f7bd55","Type":"ContainerDied","Data":"1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb"}
Jul 24 13:27:44.620041 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.620022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl" event={"ID":"0e55defe-4edd-4434-ae00-d53d23f7bd55","Type":"ContainerDied","Data":"d19bb3aa81befefdf4373b2117051b96cd338b3c344e80683783c6e7434f7718"}
Jul 24 13:27:44.620222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.620052    2579 scope.go:117] "RemoveContainer" containerID="1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb"
Jul 24 13:27:44.623473 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.623452    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.623473 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.623473    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.623660 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.623483    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e55defe-4edd-4434-ae00-d53d23f7bd55-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.623660 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.623495    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.623660 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.623504    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5btb\" (UniqueName: \"kubernetes.io/projected/0e55defe-4edd-4434-ae00-d53d23f7bd55-kube-api-access-v5btb\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.623660 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.623513    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e55defe-4edd-4434-ae00-d53d23f7bd55-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:27:44.630132 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.630115    2579 scope.go:117] "RemoveContainer" containerID="fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842"
Jul 24 13:27:44.638463 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.638445    2579 scope.go:117] "RemoveContainer" containerID="1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb"
Jul 24 13:27:44.638716 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:27:44.638699    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb\": container with ID starting with 1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb not found: ID does not exist" containerID="1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb"
Jul 24 13:27:44.638765 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.638725    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb"} err="failed to get container status \"1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb\": rpc error: code = NotFound desc = could not find container \"1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb\": container with ID starting with 1c76a2203acfd3f38ad48d62a1906c7e146163c2385257367e629028596cf8eb not found: ID does not exist"
Jul 24 13:27:44.638765 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.638742    2579 scope.go:117] "RemoveContainer" containerID="fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842"
Jul 24 13:27:44.638971 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:27:44.638950    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842\": container with ID starting with fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842 not found: ID does not exist" containerID="fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842"
Jul 24 13:27:44.639033 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.638981    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842"} err="failed to get container status \"fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842\": rpc error: code = NotFound desc = could not find container \"fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842\": container with ID starting with fb7741a4d8bc714d55fdf467f870a92f47dd8f1c9d0bb5ca70c0247a76c2e842 not found: ID does not exist"
Jul 24 13:27:44.646408 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.646385    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"]
Jul 24 13:27:44.653823 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:44.653799    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-416b8ad5-openshift-default-6987f9bdd8-z2brl"]
Jul 24 13:27:46.564369 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:27:46.564337    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" path="/var/lib/kubelet/pods/0e55defe-4edd-4434-ae00-d53d23f7bd55/volumes"
Jul 24 13:28:39.065610 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.065575    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"]
Jul 24 13:28:39.066087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.065957    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50a3d62-964f-4221-9d46-8f18c08a763e" containerName="maas-api"
Jul 24 13:28:39.066087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.065972    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50a3d62-964f-4221-9d46-8f18c08a763e" containerName="maas-api"
Jul 24 13:28:39.066087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.065983    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy"
Jul 24 13:28:39.066087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.065990    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy"
Jul 24 13:28:39.066087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.065997    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy"
Jul 24 13:28:39.066087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.066002    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy"
Jul 24 13:28:39.066087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.066014    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573dae6c-568d-4138-8bab-50cdcafbed86" containerName="main"
Jul 24 13:28:39.066087 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.066019    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="573dae6c-568d-4138-8bab-50cdcafbed86" containerName="main"
Jul 24 13:28:39.066331 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.066092    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="573dae6c-568d-4138-8bab-50cdcafbed86" containerName="main"
Jul 24 13:28:39.066331 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.066101    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50a3d62-964f-4221-9d46-8f18c08a763e" containerName="maas-api"
Jul 24 13:28:39.066331 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.066106    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy"
Jul 24 13:28:39.066331 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.066112    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e55defe-4edd-4434-ae00-d53d23f7bd55" containerName="istio-proxy"
Jul 24 13:28:39.069166 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.069148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.072315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.072296    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c72f4f-openshift-default-dockercfg-xjhcf\""
Jul 24 13:28:39.084043 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.084020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"]
Jul 24 13:28:39.198967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.198934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-certs\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.198967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.198967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istiod-ca-cert\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.199170 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.198987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-token\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.199170 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.199027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-envoy\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.199170 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.199098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-socket\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.199170 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.199133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-credential-socket\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.199297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.199176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-data\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.199297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.199202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrl4\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-kube-api-access-hnrl4\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.199297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.199234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-podinfo\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.299717 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.299678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-envoy\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.299906 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.299729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-socket\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.299906 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.299862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-credential-socket\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300076 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.299964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-data\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300076 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrl4\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-kube-api-access-hnrl4\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300076 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-podinfo\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300228 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-certs\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300228 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istiod-ca-cert\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300228 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-socket\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300228 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-token\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300423 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-data\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300423 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-credential-socket\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.300534 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.300514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-certs\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.301186 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.301165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istiod-ca-cert\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.302238 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.302220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-envoy\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.302618 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.302600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-podinfo\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.318065 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.318000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-token\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.318178 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.318161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrl4\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-kube-api-access-hnrl4\") pod \"e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") " pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.379141 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.379109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:39.516324 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.516300    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"]
Jul 24 13:28:39.517398 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:28:39.517373    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b599a4a_e6fa_4fe2_9e9e_2071cf04721a.slice/crio-d94f042334980c22030a8354f0fe8e3807d09757e5a08d28c6a8e4f5a462bf4e WatchSource:0}: Error finding container d94f042334980c22030a8354f0fe8e3807d09757e5a08d28c6a8e4f5a462bf4e: Status 404 returned error can't find the container with id d94f042334980c22030a8354f0fe8e3807d09757e5a08d28c6a8e4f5a462bf4e
Jul 24 13:28:39.519512 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.519485    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:39.519586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.519551    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:39.519586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.519582    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:39.908835 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.908800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" event={"ID":"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a","Type":"ContainerStarted","Data":"25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee"}
Jul 24 13:28:39.909014 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.908863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" event={"ID":"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a","Type":"ContainerStarted","Data":"d94f042334980c22030a8354f0fe8e3807d09757e5a08d28c6a8e4f5a462bf4e"}
Jul 24 13:28:39.936357 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:39.936312    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" podStartSLOduration=0.936297244 podStartE2EDuration="936.297244ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:28:39.93408746 +0000 UTC m=+2143.923406180" watchObservedRunningTime="2026-07-24 13:28:39.936297244 +0000 UTC m=+2143.925615939"
Jul 24 13:28:40.379299 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:40.379212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:40.383782 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:40.383757    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:40.914266 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:40.914241    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:40.915101 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:40.915080    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:42.107275 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.107230    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"]
Jul 24 13:28:42.110636 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.110610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.113896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.113873    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:28:42.114901 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.114876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:28:42.115051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.114884    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:28:42.115051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.114972    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c72f4f-serving-cert\""
Jul 24 13:28:42.127202 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.127174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"]
Jul 24 13:28:42.225299 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.225263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.225299 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.225314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhf5s\" (UniqueName: \"kubernetes.io/projected/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-kube-api-access-hhf5s\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.225536 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.225402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.326298 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.326264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.326468 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.326310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhf5s\" (UniqueName: \"kubernetes.io/projected/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-kube-api-access-hhf5s\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.326468 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.326358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.326468 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:28:42.326423    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-serving-cert: secret "maas-api-e2e-inf-a-c72f4f-serving-cert" not found
Jul 24 13:28:42.326571 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:28:42.326499    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls podName:f3a5edc0-b6e6-4e2f-8389-a63d81af94e1 nodeName:}" failed. No retries permitted until 2026-07-24 13:28:42.826482636 +0000 UTC m=+2146.815801310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls") pod "maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh" (UID: "f3a5edc0-b6e6-4e2f-8389-a63d81af94e1") : secret "maas-api-e2e-inf-a-c72f4f-serving-cert" not found
Jul 24 13:28:42.326968 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.326951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.339404 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.339384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhf5s\" (UniqueName: \"kubernetes.io/projected/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-kube-api-access-hhf5s\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.830697 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.830662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:42.833277 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:42.833253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:43.023061 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:43.023018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:43.162825 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:43.162793    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"]
Jul 24 13:28:43.167129 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:28:43.167106    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5edc0_b6e6_4e2f_8389_a63d81af94e1.slice/crio-72d2a69dc495f6d87db6ed82ba035e3d6644fa6c181314794e1e20ea2e9ef3ea WatchSource:0}: Error finding container 72d2a69dc495f6d87db6ed82ba035e3d6644fa6c181314794e1e20ea2e9ef3ea: Status 404 returned error can't find the container with id 72d2a69dc495f6d87db6ed82ba035e3d6644fa6c181314794e1e20ea2e9ef3ea
Jul 24 13:28:43.934018 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:43.933902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh" event={"ID":"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1","Type":"ContainerStarted","Data":"b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686"}
Jul 24 13:28:43.934018 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:43.933987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh" event={"ID":"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1","Type":"ContainerStarted","Data":"72d2a69dc495f6d87db6ed82ba035e3d6644fa6c181314794e1e20ea2e9ef3ea"}
Jul 24 13:28:43.934246 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:43.934117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:43.954691 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:43.954643    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh" podStartSLOduration=1.438785605 podStartE2EDuration="1.954625585s" totalImagesPullingTime="515.83998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:28:43.953720331 +0000 UTC m=+2147.943039026" watchObservedRunningTime="2026-07-24 13:28:43.954625585 +0000 UTC m=+2147.943944283"
Jul 24 13:28:47.086179 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.086100    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"]
Jul 24 13:28:47.090602 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.090576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.093370 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.093343    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b318c6-openshift-default-dockercfg-rwpwg\""
Jul 24 13:28:47.103694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.103670    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"]
Jul 24 13:28:47.167675 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.167643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-envoy\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.167842 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.167682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-podinfo\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.167842 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.167711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-certs\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.167842 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.167731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-token\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.167842 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.167822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istiod-ca-cert\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.168075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.167886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvl5\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-kube-api-access-6pvl5\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.168075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.167916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-credential-socket\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.168075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.167980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-data\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.168075 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.168053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-socket\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269379 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-envoy\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269379 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-podinfo\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269628 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-certs\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269628 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-token\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269628 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istiod-ca-cert\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269628 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvl5\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-kube-api-access-6pvl5\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269628 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-credential-socket\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269628 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-data\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.269628 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-socket\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.270023 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-certs\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.270023 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.269970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-credential-socket\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.270092 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.270043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-socket\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.270139 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.270087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-data\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.270526 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.270503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istiod-ca-cert\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.271906 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.271888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-envoy\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.272072 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.272055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-podinfo\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.286388 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.286366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvl5\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-kube-api-access-6pvl5\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.286631 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.286615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-token\") pod \"e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") " pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.402385 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.402302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:47.541175 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.541144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"]
Jul 24 13:28:47.542401 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:28:47.542370    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fa5a8d_5c0d_4b90_ba1b_2c8e1f18b293.slice/crio-e2e31b2fa3cebffcc07e1202cd269cd661b020e92a0c0545ff629435f4d903fb WatchSource:0}: Error finding container e2e31b2fa3cebffcc07e1202cd269cd661b020e92a0c0545ff629435f4d903fb: Status 404 returned error can't find the container with id e2e31b2fa3cebffcc07e1202cd269cd661b020e92a0c0545ff629435f4d903fb
Jul 24 13:28:47.544570 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.544535    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:47.544631 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.544602    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:47.544674 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.544629    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:47.956545 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.956496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" event={"ID":"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293","Type":"ContainerStarted","Data":"db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867"}
Jul 24 13:28:47.956728 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.956574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" event={"ID":"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293","Type":"ContainerStarted","Data":"e2e31b2fa3cebffcc07e1202cd269cd661b020e92a0c0545ff629435f4d903fb"}
Jul 24 13:28:47.983618 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:47.983572    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" podStartSLOduration=0.983557236 podStartE2EDuration="983.557236ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:28:47.980663817 +0000 UTC m=+2151.969982513" watchObservedRunningTime="2026-07-24 13:28:47.983557236 +0000 UTC m=+2151.972875930"
Jul 24 13:28:48.403073 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:48.402877    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:48.413734 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:48.413693    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:48.961965 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:48.961934    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:48.962876 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:48.962857    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:28:49.944956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:49.944906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:28:50.082945 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.082902    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"]
Jul 24 13:28:50.086699 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.086679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.094203 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.093986    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b318c6-serving-cert\""
Jul 24 13:28:50.106948 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.103510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.106948 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.103560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e579340-2562-4647-8912-d45e76809f89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.106948 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.103663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kvg\" (UniqueName: \"kubernetes.io/projected/7e579340-2562-4647-8912-d45e76809f89-kube-api-access-25kvg\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.109448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.109421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"]
Jul 24 13:28:50.204589 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.204463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kvg\" (UniqueName: \"kubernetes.io/projected/7e579340-2562-4647-8912-d45e76809f89-kube-api-access-25kvg\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.205335 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.205294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.205683 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.205666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e579340-2562-4647-8912-d45e76809f89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.206864 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.206847    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e579340-2562-4647-8912-d45e76809f89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.207038 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:28:50.205612    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-serving-cert: secret "maas-api-e2e-inf-b-b318c6-serving-cert" not found
Jul 24 13:28:50.207219 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:28:50.207194    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls podName:7e579340-2562-4647-8912-d45e76809f89 nodeName:}" failed. No retries permitted until 2026-07-24 13:28:50.707171229 +0000 UTC m=+2154.696489907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls") pod "maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z" (UID: "7e579340-2562-4647-8912-d45e76809f89") : secret "maas-api-e2e-inf-b-b318c6-serving-cert" not found
Jul 24 13:28:50.224608 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.224570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kvg\" (UniqueName: \"kubernetes.io/projected/7e579340-2562-4647-8912-d45e76809f89-kube-api-access-25kvg\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.711569 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.711537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.717402 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.717379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls\") pod \"maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.719707 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.719687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:50.921146 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.920228    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"]
Jul 24 13:28:50.973220 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:50.973136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z" event={"ID":"7e579340-2562-4647-8912-d45e76809f89","Type":"ContainerStarted","Data":"a186bf10ae9199eba23e51edef24954c958d51d6e2a73d80d6f092214b42fad2"}
Jul 24 13:28:51.980675 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:51.980640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z" event={"ID":"7e579340-2562-4647-8912-d45e76809f89","Type":"ContainerStarted","Data":"3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a"}
Jul 24 13:28:51.981131 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:51.980881    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:52.005289 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:52.005225    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z" podStartSLOduration=1.5583793529999999 podStartE2EDuration="2.005205169s" totalImagesPullingTime="446.825816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:28:52.003421151 +0000 UTC m=+2155.992739848" watchObservedRunningTime="2026-07-24 13:28:52.005205169 +0000 UTC m=+2155.994523865"
Jul 24 13:28:55.915808 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:55.915765    2579 patch_prober.go:28] interesting pod/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 13:28:55.916274 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:55.915835    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 13:28:56.016405 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:56.016357    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerID="25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee" exitCode=0
Jul 24 13:28:56.016600 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:56.016470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" event={"ID":"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a","Type":"ContainerDied","Data":"25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee"}
Jul 24 13:28:56.017152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:56.017123    2579 scope.go:117] "RemoveContainer" containerID="25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee"
Jul 24 13:28:56.018848 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:56.018798    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:56.018988 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:56.018967    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:56.019078 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:56.019062    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:28:57.027679 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:57.027643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" event={"ID":"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a","Type":"ContainerStarted","Data":"8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768"}
Jul 24 13:28:57.379987 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:57.379896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:28:57.380211 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:57.380187    2579 patch_prober.go:28] interesting pod/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 13:28:57.380263 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:57.380236    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 13:28:58.014272 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:58.014235    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:28:58.380469 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:58.380355    2579 patch_prober.go:28] interesting pod/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 13:28:58.380469 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:58.380435    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 13:28:59.379680 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:59.379584    2579 patch_prober.go:28] interesting pod/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 13:28:59.379856 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:28:59.379677    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 13:29:00.379646 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:00.379611    2579 patch_prober.go:28] interesting pod/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 13:29:00.380062 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:00.379670    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 13:29:01.384211 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:01.384181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:29:01.384676 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:01.384653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:29:01.385390 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:01.385369    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:29:02.388375 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:29:02.388345    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fa5a8d_5c0d_4b90_ba1b_2c8e1f18b293.slice/crio-conmon-db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867.scope\": RecentStats: unable to find data in memory cache]"
Jul 24 13:29:03.069626 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:03.069592    2579 generic.go:356] "Generic (PLEG): container finished" podID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerID="db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867" exitCode=0
Jul 24 13:29:03.069819 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:03.069666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" event={"ID":"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293","Type":"ContainerDied","Data":"db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867"}
Jul 24 13:29:03.070136 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:03.070121    2579 scope.go:117] "RemoveContainer" containerID="db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867"
Jul 24 13:29:03.071118 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:03.071097    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:29:03.071175 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:03.071155    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:29:03.071214 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:03.071185    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:29:03.962740 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:03.962700    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:29:04.077863 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:04.077831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" event={"ID":"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293","Type":"ContainerStarted","Data":"a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1"}
Jul 24 13:29:04.403200 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:04.403054    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:29:04.404516 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:04.404490    2579 patch_prober.go:28] interesting pod/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 24 13:29:04.404723 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:04.404699    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 24 13:29:05.402897 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:05.402858    2579 patch_prober.go:28] interesting pod/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 24 13:29:05.403488 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:05.403457    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 24 13:29:06.403771 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:06.403668    2579 patch_prober.go:28] interesting pod/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 24 13:29:06.403771 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:06.403736    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 24 13:29:07.402968 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:07.402903    2579 patch_prober.go:28] interesting pod/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 24 13:29:07.403204 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:07.402994    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 24 13:29:08.406820 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:08.406791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:29:08.407264 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:08.407196    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:29:08.407767 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:08.407751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:29:32.104710 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.104678    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"]
Jul 24 13:29:32.107685 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.107665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.111347 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.111324    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b318c6\"/\"default-dockercfg-bscg9\""
Jul 24 13:29:32.111347 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.111339    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b318c6\"/\"kube-root-ca.crt\""
Jul 24 13:29:32.111529 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.111394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b318c6\"/\"test-model-b318c6-kserve-self-signed-certs\""
Jul 24 13:29:32.111529 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.111444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b318c6\"/\"openshift-service-ca.crt\""
Jul 24 13:29:32.117440 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.117413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tls-certs\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.117576 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.117455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjk6f\" (UniqueName: \"kubernetes.io/projected/5fc41ef1-bf19-4dc8-b743-9498e1543e68-kube-api-access-qjk6f\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.117576 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.117486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-dshm\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.117576 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.117550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-model-cache\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.117768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.117589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-home\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.117768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.117608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tmp-dir\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.119457 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.119427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"]
Jul 24 13:29:32.218765 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.218734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-dshm\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.218956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.218788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-model-cache\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.218956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.218814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-home\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.218956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.218829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tmp-dir\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.218956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.218884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tls-certs\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.218956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.218912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjk6f\" (UniqueName: \"kubernetes.io/projected/5fc41ef1-bf19-4dc8-b743-9498e1543e68-kube-api-access-qjk6f\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.219266 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.219240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-model-cache\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.219384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.219275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-home\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.219384 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.219308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tmp-dir\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.221135 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.221110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-dshm\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.221414 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.221396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tls-certs\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.229111 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.229091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjk6f\" (UniqueName: \"kubernetes.io/projected/5fc41ef1-bf19-4dc8-b743-9498e1543e68-kube-api-access-qjk6f\") pod \"test-model-b318c6-kserve-6bc6f4c94c-lxvm4\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") " pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.418853 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.418822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:32.564232 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:29:32.564204    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fc41ef1_bf19_4dc8_b743_9498e1543e68.slice/crio-249224947f71e2356618e779523b2db6fd888f2330fb5cae22c2005e5845d6b3 WatchSource:0}: Error finding container 249224947f71e2356618e779523b2db6fd888f2330fb5cae22c2005e5845d6b3: Status 404 returned error can't find the container with id 249224947f71e2356618e779523b2db6fd888f2330fb5cae22c2005e5845d6b3
Jul 24 13:29:32.564479 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:32.564455    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"]
Jul 24 13:29:33.241121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:33.241081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4" event={"ID":"5fc41ef1-bf19-4dc8-b743-9498e1543e68","Type":"ContainerStarted","Data":"503ee32453a0c39aab3e55a8be8c30e1033408a3284b0886a40e3266a8eec5ea"}
Jul 24 13:29:33.241468 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:33.241135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4" event={"ID":"5fc41ef1-bf19-4dc8-b743-9498e1543e68","Type":"ContainerStarted","Data":"249224947f71e2356618e779523b2db6fd888f2330fb5cae22c2005e5845d6b3"}
Jul 24 13:29:33.260394 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:33.260347    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4" podStartSLOduration=1.260332371 podStartE2EDuration="1.260332371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:29:33.2584703 +0000 UTC m=+2197.247788994" watchObservedRunningTime="2026-07-24 13:29:33.260332371 +0000 UTC m=+2197.249651066"
Jul 24 13:29:42.419811 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:42.419777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:42.419811 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:42.419816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:42.432377 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:42.432347    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:29:43.307049 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:29:43.307017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:30:15.373552 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:15.373517    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"]
Jul 24 13:30:26.218555 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.218481    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"]
Jul 24 13:30:26.218937 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.218707    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh" podUID="f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" containerName="maas-api" containerID="cri-o://b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686" gracePeriod=30
Jul 24 13:30:26.468225 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.468079    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:30:26.509680 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.509587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhf5s\" (UniqueName: \"kubernetes.io/projected/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-kube-api-access-hhf5s\") pod \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") "
Jul 24 13:30:26.509680 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.509654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-openshift-service-ca\") pod \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") "
Jul 24 13:30:26.509907 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.509744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls\") pod \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\" (UID: \"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1\") "
Jul 24 13:30:26.510080 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.510045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" (UID: "f3a5edc0-b6e6-4e2f-8389-a63d81af94e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:30:26.512022 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.511982    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-kube-api-access-hhf5s" (OuterVolumeSpecName: "kube-api-access-hhf5s") pod "f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" (UID: "f3a5edc0-b6e6-4e2f-8389-a63d81af94e1"). InnerVolumeSpecName "kube-api-access-hhf5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:30:26.512148 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.512122    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" (UID: "f3a5edc0-b6e6-4e2f-8389-a63d81af94e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:30:26.531241 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.531209    2579 generic.go:356] "Generic (PLEG): container finished" podID="f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" containerID="b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686" exitCode=0
Jul 24 13:30:26.531334 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.531254    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"
Jul 24 13:30:26.531334 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.531278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh" event={"ID":"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1","Type":"ContainerDied","Data":"b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686"}
Jul 24 13:30:26.531334 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.531315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh" event={"ID":"f3a5edc0-b6e6-4e2f-8389-a63d81af94e1","Type":"ContainerDied","Data":"72d2a69dc495f6d87db6ed82ba035e3d6644fa6c181314794e1e20ea2e9ef3ea"}
Jul 24 13:30:26.531334 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.531335    2579 scope.go:117] "RemoveContainer" containerID="b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686"
Jul 24 13:30:26.542415 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.542395    2579 scope.go:117] "RemoveContainer" containerID="b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686"
Jul 24 13:30:26.542688 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:30:26.542670    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686\": container with ID starting with b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686 not found: ID does not exist" containerID="b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686"
Jul 24 13:30:26.542768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.542695    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686"} err="failed to get container status \"b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686\": rpc error: code = NotFound desc = could not find container \"b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686\": container with ID starting with b8bb7eeb344b5b769e271c5c01e56707e8703540a9d57468dfea2a1375f23686 not found: ID does not exist"
Jul 24 13:30:26.569970 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.569943    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"]
Jul 24 13:30:26.569970 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.569972    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c72f4f-7cb5f46848-q97gh"]
Jul 24 13:30:26.611580 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.611556    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:26.611580 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.611574    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhf5s\" (UniqueName: \"kubernetes.io/projected/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-kube-api-access-hhf5s\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:26.611580 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:26.611584    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:28.562267 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:28.562235    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" path="/var/lib/kubelet/pods/f3a5edc0-b6e6-4e2f-8389-a63d81af94e1/volumes"
Jul 24 13:30:30.395867 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.395812    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4" podUID="5fc41ef1-bf19-4dc8-b743-9498e1543e68" containerName="main" containerID="cri-o://503ee32453a0c39aab3e55a8be8c30e1033408a3284b0886a40e3266a8eec5ea" gracePeriod=45
Jul 24 13:30:30.555169 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.555132    2579 generic.go:356] "Generic (PLEG): container finished" podID="5fc41ef1-bf19-4dc8-b743-9498e1543e68" containerID="503ee32453a0c39aab3e55a8be8c30e1033408a3284b0886a40e3266a8eec5ea" exitCode=0
Jul 24 13:30:30.555348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.555188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4" event={"ID":"5fc41ef1-bf19-4dc8-b743-9498e1543e68","Type":"ContainerDied","Data":"503ee32453a0c39aab3e55a8be8c30e1033408a3284b0886a40e3266a8eec5ea"}
Jul 24 13:30:30.654972 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.654950    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:30:30.749705 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.749679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tls-certs\") pod \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") "
Jul 24 13:30:30.749868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.749720    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjk6f\" (UniqueName: \"kubernetes.io/projected/5fc41ef1-bf19-4dc8-b743-9498e1543e68-kube-api-access-qjk6f\") pod \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") "
Jul 24 13:30:30.749868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.749768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tmp-dir\") pod \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") "
Jul 24 13:30:30.749868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.749805    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-dshm\") pod \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") "
Jul 24 13:30:30.749868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.749826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-model-cache\") pod \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") "
Jul 24 13:30:30.749868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.749854    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-home\") pod \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\" (UID: \"5fc41ef1-bf19-4dc8-b743-9498e1543e68\") "
Jul 24 13:30:30.750145 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.750122    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fc41ef1-bf19-4dc8-b743-9498e1543e68" (UID: "5fc41ef1-bf19-4dc8-b743-9498e1543e68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:30.750206 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.750151    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-home" (OuterVolumeSpecName: "home") pod "5fc41ef1-bf19-4dc8-b743-9498e1543e68" (UID: "5fc41ef1-bf19-4dc8-b743-9498e1543e68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:30.750206 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.750168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fc41ef1-bf19-4dc8-b743-9498e1543e68" (UID: "5fc41ef1-bf19-4dc8-b743-9498e1543e68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:30.750304 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.750221    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-model-cache\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:30.752187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.752165    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-dshm" (OuterVolumeSpecName: "dshm") pod "5fc41ef1-bf19-4dc8-b743-9498e1543e68" (UID: "5fc41ef1-bf19-4dc8-b743-9498e1543e68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:30.752187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.752180    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fc41ef1-bf19-4dc8-b743-9498e1543e68" (UID: "5fc41ef1-bf19-4dc8-b743-9498e1543e68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:30:30.752326 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.752249    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc41ef1-bf19-4dc8-b743-9498e1543e68-kube-api-access-qjk6f" (OuterVolumeSpecName: "kube-api-access-qjk6f") pod "5fc41ef1-bf19-4dc8-b743-9498e1543e68" (UID: "5fc41ef1-bf19-4dc8-b743-9498e1543e68"). InnerVolumeSpecName "kube-api-access-qjk6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:30:30.851671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.851646    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tmp-dir\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:30.851671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.851669    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-dshm\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:30.851671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.851677    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc41ef1-bf19-4dc8-b743-9498e1543e68-home\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:30.851852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.851684    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc41ef1-bf19-4dc8-b743-9498e1543e68-tls-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:30.851852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:30.851693    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjk6f\" (UniqueName: \"kubernetes.io/projected/5fc41ef1-bf19-4dc8-b743-9498e1543e68-kube-api-access-qjk6f\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:31.562282 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:31.562254    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"
Jul 24 13:30:31.562282 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:31.562273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4" event={"ID":"5fc41ef1-bf19-4dc8-b743-9498e1543e68","Type":"ContainerDied","Data":"249224947f71e2356618e779523b2db6fd888f2330fb5cae22c2005e5845d6b3"}
Jul 24 13:30:31.562806 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:31.562310    2579 scope.go:117] "RemoveContainer" containerID="503ee32453a0c39aab3e55a8be8c30e1033408a3284b0886a40e3266a8eec5ea"
Jul 24 13:30:31.587862 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:31.587839    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"]
Jul 24 13:30:31.596630 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:31.596606    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b318c6/test-model-b318c6-kserve-6bc6f4c94c-lxvm4"]
Jul 24 13:30:32.570698 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:32.570660    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fc41ef1-bf19-4dc8-b743-9498e1543e68" path="/var/lib/kubelet/pods/5fc41ef1-bf19-4dc8-b743-9498e1543e68/volumes"
Jul 24 13:30:38.246434 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:38.246394    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"]
Jul 24 13:30:38.246951 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:38.246613    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy" containerID="cri-o://8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768" gracePeriod=30
Jul 24 13:30:43.501599 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.501575    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:30:43.577624 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnrl4\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-kube-api-access-hnrl4\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.577624 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-podinfo\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.577624 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-token\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.577896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577639    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-socket\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.577896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-credential-socket\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.577896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-envoy\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.577896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-data\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.577896 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:43.578207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istiod-ca-cert\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.578207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.577987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-certs\") pod \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\" (UID: \"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a\") "
Jul 24 13:30:43.578207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.578038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:43.578363 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.578322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:30:43.578584 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.578502    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:43.578584 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.578522    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:43.578584 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.578520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:43.578744 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.578598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:43.580012 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.579984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:30:43.580124 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.580028    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-kube-api-access-hnrl4" (OuterVolumeSpecName: "kube-api-access-hnrl4") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "kube-api-access-hnrl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:30:43.580124 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.580094    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:30:43.580309 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.580293    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" (UID: "4b599a4a-e6fa-4fe2-9e9e-2071cf04721a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:30:43.634699 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.634671    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerID="8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768" exitCode=0
Jul 24 13:30:43.634847 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.634717    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"
Jul 24 13:30:43.634847 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.634738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" event={"ID":"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a","Type":"ContainerDied","Data":"8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768"}
Jul 24 13:30:43.634847 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.634792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r" event={"ID":"4b599a4a-e6fa-4fe2-9e9e-2071cf04721a","Type":"ContainerDied","Data":"d94f042334980c22030a8354f0fe8e3807d09757e5a08d28c6a8e4f5a462bf4e"}
Jul 24 13:30:43.634847 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.634814    2579 scope.go:117] "RemoveContainer" containerID="8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768"
Jul 24 13:30:43.644638 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.644622    2579 scope.go:117] "RemoveContainer" containerID="25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee"
Jul 24 13:30:43.652761 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.652745    2579 scope.go:117] "RemoveContainer" containerID="8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768"
Jul 24 13:30:43.653013 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:30:43.652995    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768\": container with ID starting with 8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768 not found: ID does not exist" containerID="8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768"
Jul 24 13:30:43.653063 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.653022    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768"} err="failed to get container status \"8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768\": rpc error: code = NotFound desc = could not find container \"8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768\": container with ID starting with 8cc22a8c2caafcf9ff758dcb41d672a685e6906e2a87c4cecb708b5b8914f768 not found: ID does not exist"
Jul 24 13:30:43.653063 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.653039    2579 scope.go:117] "RemoveContainer" containerID="25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee"
Jul 24 13:30:43.653289 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:30:43.653272    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee\": container with ID starting with 25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee not found: ID does not exist" containerID="25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee"
Jul 24 13:30:43.653331 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.653296    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee"} err="failed to get container status \"25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee\": rpc error: code = NotFound desc = could not find container \"25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee\": container with ID starting with 25c49cbd924b2c280e95127ec303cb4beaf86ece158f7a7ea9f3a29c44a996ee not found: ID does not exist"
Jul 24 13:30:43.662109 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.662083    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"]
Jul 24 13:30:43.668401 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.668382    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c72f4f-openshift-default-bb5cf8df4-db48r"]
Jul 24 13:30:43.679337 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.679318    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:43.679419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.679340    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:43.679419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.679353    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:43.679419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.679363    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:43.679419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.679375    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnrl4\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-kube-api-access-hnrl4\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:43.679419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.679390    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:43.679419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:43.679402    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:44.563695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:44.563662    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" path="/var/lib/kubelet/pods/4b599a4a-e6fa-4fe2-9e9e-2071cf04721a/volumes"
Jul 24 13:30:49.180337 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.180301    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"]
Jul 24 13:30:49.180812 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.180540    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z" podUID="7e579340-2562-4647-8912-d45e76809f89" containerName="maas-api" containerID="cri-o://3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a" gracePeriod=30
Jul 24 13:30:49.437971 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.437878    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:30:49.529570 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.529543    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25kvg\" (UniqueName: \"kubernetes.io/projected/7e579340-2562-4647-8912-d45e76809f89-kube-api-access-25kvg\") pod \"7e579340-2562-4647-8912-d45e76809f89\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") "
Jul 24 13:30:49.529741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.529674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls\") pod \"7e579340-2562-4647-8912-d45e76809f89\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") "
Jul 24 13:30:49.529741 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.529696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e579340-2562-4647-8912-d45e76809f89-openshift-service-ca\") pod \"7e579340-2562-4647-8912-d45e76809f89\" (UID: \"7e579340-2562-4647-8912-d45e76809f89\") "
Jul 24 13:30:49.530147 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.530121    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e579340-2562-4647-8912-d45e76809f89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e579340-2562-4647-8912-d45e76809f89" (UID: "7e579340-2562-4647-8912-d45e76809f89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:30:49.531749 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.531722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e579340-2562-4647-8912-d45e76809f89" (UID: "7e579340-2562-4647-8912-d45e76809f89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:30:49.531856 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.531800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e579340-2562-4647-8912-d45e76809f89-kube-api-access-25kvg" (OuterVolumeSpecName: "kube-api-access-25kvg") pod "7e579340-2562-4647-8912-d45e76809f89" (UID: "7e579340-2562-4647-8912-d45e76809f89"). InnerVolumeSpecName "kube-api-access-25kvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:30:49.631511 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.631467    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e579340-2562-4647-8912-d45e76809f89-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:49.631511 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.631502    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e579340-2562-4647-8912-d45e76809f89-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:49.631511 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.631517    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kvg\" (UniqueName: \"kubernetes.io/projected/7e579340-2562-4647-8912-d45e76809f89-kube-api-access-25kvg\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:30:49.672025 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.671995    2579 generic.go:356] "Generic (PLEG): container finished" podID="7e579340-2562-4647-8912-d45e76809f89" containerID="3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a" exitCode=0
Jul 24 13:30:49.672174 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.672051    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"
Jul 24 13:30:49.672174 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.672079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z" event={"ID":"7e579340-2562-4647-8912-d45e76809f89","Type":"ContainerDied","Data":"3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a"}
Jul 24 13:30:49.672174 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.672132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z" event={"ID":"7e579340-2562-4647-8912-d45e76809f89","Type":"ContainerDied","Data":"a186bf10ae9199eba23e51edef24954c958d51d6e2a73d80d6f092214b42fad2"}
Jul 24 13:30:49.672174 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.672155    2579 scope.go:117] "RemoveContainer" containerID="3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a"
Jul 24 13:30:49.681624 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.681611    2579 scope.go:117] "RemoveContainer" containerID="3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a"
Jul 24 13:30:49.681881 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:30:49.681855    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a\": container with ID starting with 3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a not found: ID does not exist" containerID="3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a"
Jul 24 13:30:49.681943 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.681887    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a"} err="failed to get container status \"3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a\": rpc error: code = NotFound desc = could not find container \"3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a\": container with ID starting with 3eb223e73a9e6ecbb634a7f7538880b9aefb35f0288dbc9079a9b8a2ce7d0c7a not found: ID does not exist"
Jul 24 13:30:49.695719 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.695664    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"]
Jul 24 13:30:49.700861 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:49.700840    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b318c6-8668b4dccb-gfc5z"]
Jul 24 13:30:50.565089 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:30:50.565053    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e579340-2562-4647-8912-d45e76809f89" path="/var/lib/kubelet/pods/7e579340-2562-4647-8912-d45e76809f89/volumes"
Jul 24 13:31:00.847688 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:00.847654    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"]
Jul 24 13:31:00.848191 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:00.847890    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy" containerID="cri-o://a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1" gracePeriod=30
Jul 24 13:31:01.830763 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.830730    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"]
Jul 24 13:31:01.831150 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831134    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831151    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831163    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" containerName="maas-api"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831169    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" containerName="maas-api"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831181    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831189    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831196    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc41ef1-bf19-4dc8-b743-9498e1543e68" containerName="main"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831201    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc41ef1-bf19-4dc8-b743-9498e1543e68" containerName="main"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831212    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e579340-2562-4647-8912-d45e76809f89" containerName="maas-api"
Jul 24 13:31:01.831221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831218    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e579340-2562-4647-8912-d45e76809f89" containerName="maas-api"
Jul 24 13:31:01.831494 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831279    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a5edc0-b6e6-4e2f-8389-a63d81af94e1" containerName="maas-api"
Jul 24 13:31:01.831494 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831289    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e579340-2562-4647-8912-d45e76809f89" containerName="maas-api"
Jul 24 13:31:01.831494 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831294    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fc41ef1-bf19-4dc8-b743-9498e1543e68" containerName="main"
Jul 24 13:31:01.831494 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831300    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy"
Jul 24 13:31:01.831494 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.831306    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b599a4a-e6fa-4fe2-9e9e-2071cf04721a" containerName="istio-proxy"
Jul 24 13:31:01.834386 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.834371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.837196 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.837180    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e530f2-openshift-default-dockercfg-5xp9z\""
Jul 24 13:31:01.849604 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.849580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"]
Jul 24 13:31:01.942713 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.942678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-credential-socket\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.942713 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.942717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-certs\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.943070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.942758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c537101-73f8-4092-9e96-bb1046826563-istiod-ca-cert\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.943070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.942780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c537101-73f8-4092-9e96-bb1046826563-istio-podinfo\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.943070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.942890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-istio-token\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.943070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.942967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztg6f\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-kube-api-access-ztg6f\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.943070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.943015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-envoy\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.943070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.943044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-socket\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:01.943288 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:01.943072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-data\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.043994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztg6f\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-kube-api-access-ztg6f\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044225 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-envoy\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044225 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-socket\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044225 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-data\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044225 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-credential-socket\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044225 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-certs\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044225 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c537101-73f8-4092-9e96-bb1046826563-istiod-ca-cert\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044225 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c537101-73f8-4092-9e96-bb1046826563-istio-podinfo\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044543 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-istio-token\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044616 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-certs\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044678 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-credential-socket\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044678 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-data\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044756 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-socket\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.044911 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.044886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c537101-73f8-4092-9e96-bb1046826563-istiod-ca-cert\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.046711 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.046686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-envoy\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.046822 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.046810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c537101-73f8-4092-9e96-bb1046826563-istio-podinfo\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.053543 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.053519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-istio-token\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.053661 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.053643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztg6f\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-kube-api-access-ztg6f\") pod \"e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") " pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.144177 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.144079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:02.289167 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.289140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"]
Jul 24 13:31:02.291155 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:31:02.291127    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c537101_73f8_4092_9e96_bb1046826563.slice/crio-05097041075e384c74e84d42bdbfd83821c16502af2c7a2666ece73a7992757e WatchSource:0}: Error finding container 05097041075e384c74e84d42bdbfd83821c16502af2c7a2666ece73a7992757e: Status 404 returned error can't find the container with id 05097041075e384c74e84d42bdbfd83821c16502af2c7a2666ece73a7992757e
Jul 24 13:31:02.293848 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.293809    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:02.293981 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.293897    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:02.293981 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.293966    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:02.742549 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.742506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" event={"ID":"2c537101-73f8-4092-9e96-bb1046826563","Type":"ContainerStarted","Data":"5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41"}
Jul 24 13:31:02.742701 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.742559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" event={"ID":"2c537101-73f8-4092-9e96-bb1046826563","Type":"ContainerStarted","Data":"05097041075e384c74e84d42bdbfd83821c16502af2c7a2666ece73a7992757e"}
Jul 24 13:31:02.768774 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:02.768729    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" podStartSLOduration=1.768716435 podStartE2EDuration="1.768716435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:02.765888779 +0000 UTC m=+2286.755207489" watchObservedRunningTime="2026-07-24 13:31:02.768716435 +0000 UTC m=+2286.758035130"
Jul 24 13:31:03.145179 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:03.145094    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:03.150610 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:03.150584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:03.748788 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:03.748676    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:03.753772 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:03.753742    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:04.969254 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.969222    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"]
Jul 24 13:31:04.972760 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.972741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:04.974275 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.974221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlq5\" (UniqueName: \"kubernetes.io/projected/2c690e6e-e9db-4788-95b3-82de2252a2de-kube-api-access-lxlq5\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:04.974402 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.974294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c690e6e-e9db-4788-95b3-82de2252a2de-openshift-service-ca\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:04.974402 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.974361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c690e6e-e9db-4788-95b3-82de2252a2de-maas-api-tls\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:04.975714 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.975695    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:31:04.975812 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.975750    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e530f2-serving-cert\""
Jul 24 13:31:04.975812 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.975769    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:31:04.976589 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.976573    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:31:04.986333 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:04.986312    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"]
Jul 24 13:31:05.075147 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.075117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlq5\" (UniqueName: \"kubernetes.io/projected/2c690e6e-e9db-4788-95b3-82de2252a2de-kube-api-access-lxlq5\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:05.075297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.075168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c690e6e-e9db-4788-95b3-82de2252a2de-openshift-service-ca\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:05.075297 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.075230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c690e6e-e9db-4788-95b3-82de2252a2de-maas-api-tls\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:05.075897 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.075865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c690e6e-e9db-4788-95b3-82de2252a2de-openshift-service-ca\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:05.077631 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.077613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c690e6e-e9db-4788-95b3-82de2252a2de-maas-api-tls\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:05.085853 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.085830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlq5\" (UniqueName: \"kubernetes.io/projected/2c690e6e-e9db-4788-95b3-82de2252a2de-kube-api-access-lxlq5\") pod \"maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:05.285577 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.285492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:05.425080 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.425054    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"]
Jul 24 13:31:05.426052 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:31:05.426027    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c690e6e_e9db_4788_95b3_82de2252a2de.slice/crio-2a7a743e53f1382c681fd83479a381bf34e83223c69ec90bdad0f203bfff4bc2 WatchSource:0}: Error finding container 2a7a743e53f1382c681fd83479a381bf34e83223c69ec90bdad0f203bfff4bc2: Status 404 returned error can't find the container with id 2a7a743e53f1382c681fd83479a381bf34e83223c69ec90bdad0f203bfff4bc2
Jul 24 13:31:05.427546 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.427532    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:31:05.761323 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:05.761279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj" event={"ID":"2c690e6e-e9db-4788-95b3-82de2252a2de","Type":"ContainerStarted","Data":"2a7a743e53f1382c681fd83479a381bf34e83223c69ec90bdad0f203bfff4bc2"}
Jul 24 13:31:06.127410 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.127391    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:31:06.186538 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186509    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-envoy\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.186538 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-certs\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.186731 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-credential-socket\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.186731 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-data\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.186731 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186623    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pvl5\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-kube-api-access-6pvl5\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.186731 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-podinfo\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.186731 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186682    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-socket\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.186731 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-token\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.187077 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istiod-ca-cert\") pod \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\" (UID: \"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293\") "
Jul 24 13:31:06.187077 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:06.187077 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.186918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:06.187436 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.187087    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.187436 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.187110    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.187436 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.187158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:31:06.187436 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.187241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:06.187436 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.187420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-data" (OuterVolumeSpecName: "istio-data") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:06.188802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.188781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:06.189019 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.188998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:31:06.189443 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.189422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-token" (OuterVolumeSpecName: "istio-token") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:31:06.189530 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.189458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-kube-api-access-6pvl5" (OuterVolumeSpecName: "kube-api-access-6pvl5") pod "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" (UID: "00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293"). InnerVolumeSpecName "kube-api-access-6pvl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:31:06.287596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.287530    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.287596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.287556    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.287596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.287564    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.287596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.287572    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.287596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.287581    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pvl5\" (UniqueName: \"kubernetes.io/projected/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-kube-api-access-6pvl5\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.287596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.287591    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.287596 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.287599    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:06.767890 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.767856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj" event={"ID":"2c690e6e-e9db-4788-95b3-82de2252a2de","Type":"ContainerStarted","Data":"bce5988f5517e3dc8113342167daa82a8900d68027b274b0750ad187a28e82d3"}
Jul 24 13:31:06.768054 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.767948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:06.769510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.769482    2579 generic.go:356] "Generic (PLEG): container finished" podID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerID="a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1" exitCode=0
Jul 24 13:31:06.769620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.769521    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"
Jul 24 13:31:06.769620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.769529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" event={"ID":"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293","Type":"ContainerDied","Data":"a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1"}
Jul 24 13:31:06.769620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.769579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw" event={"ID":"00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293","Type":"ContainerDied","Data":"e2e31b2fa3cebffcc07e1202cd269cd661b020e92a0c0545ff629435f4d903fb"}
Jul 24 13:31:06.769620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.769603    2579 scope.go:117] "RemoveContainer" containerID="a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1"
Jul 24 13:31:06.778301 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.778286    2579 scope.go:117] "RemoveContainer" containerID="db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867"
Jul 24 13:31:06.786271 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.786255    2579 scope.go:117] "RemoveContainer" containerID="a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1"
Jul 24 13:31:06.786522 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:31:06.786504    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1\": container with ID starting with a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1 not found: ID does not exist" containerID="a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1"
Jul 24 13:31:06.786589 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.786529    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1"} err="failed to get container status \"a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1\": rpc error: code = NotFound desc = could not find container \"a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1\": container with ID starting with a075c0407a82d181db570cdeacac2dd7576dfe41dc20c543cd27036abba088a1 not found: ID does not exist"
Jul 24 13:31:06.786589 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.786577    2579 scope.go:117] "RemoveContainer" containerID="db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867"
Jul 24 13:31:06.786913 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:31:06.786879    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867\": container with ID starting with db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867 not found: ID does not exist" containerID="db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867"
Jul 24 13:31:06.787120 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.786913    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867"} err="failed to get container status \"db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867\": rpc error: code = NotFound desc = could not find container \"db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867\": container with ID starting with db6fd4c947219e0b4b634e97384f08a189a1cf35fc7c878fc1d7c0d983132867 not found: ID does not exist"
Jul 24 13:31:06.789193 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.789149    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj" podStartSLOduration=2.146075325 podStartE2EDuration="2.789137559s" totalImagesPullingTime="643.062234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:06.788239894 +0000 UTC m=+2290.777558607" watchObservedRunningTime="2026-07-24 13:31:06.789137559 +0000 UTC m=+2290.778456255"
Jul 24 13:31:06.805315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.805294    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"]
Jul 24 13:31:06.810104 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:06.810084    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b318c6-openshift-default-5bb86999c8-8b6zw"]
Jul 24 13:31:08.562935 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:08.562888    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" path="/var/lib/kubelet/pods/00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293/volumes"
Jul 24 13:31:10.218726 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.218679    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"]
Jul 24 13:31:10.220035 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.219993    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy"
Jul 24 13:31:10.222144 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.222125    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy"
Jul 24 13:31:10.222303 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.222289    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy"
Jul 24 13:31:10.222395 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.222385    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy"
Jul 24 13:31:10.222597 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.222584    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy"
Jul 24 13:31:10.227569 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.227548    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"]
Jul 24 13:31:10.227647 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.227640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.230602 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.230579    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a9463a-openshift-default-dockercfg-wk7fb\""
Jul 24 13:31:10.323063 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-token\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.323260 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c0f8d6-e595-4446-81be-b5891c8aae79-istiod-ca-cert\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.323260 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-podinfo\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.323260 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-socket\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.323260 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-certs\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.323417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-credential-socket\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.323417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-envoy\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.323417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvllg\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-kube-api-access-qvllg\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.323417 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.323390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-data\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424117 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-credential-socket\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424321 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-envoy\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424321 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvllg\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-kube-api-access-qvllg\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424321 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-data\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-token\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c0f8d6-e595-4446-81be-b5891c8aae79-istiod-ca-cert\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-podinfo\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-socket\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424703 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-credential-socket\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424703 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-certs\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424703 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-data\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424816 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-socket\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.424905 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.424887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-certs\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.425215 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.425199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c0f8d6-e595-4446-81be-b5891c8aae79-istiod-ca-cert\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.426550 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.426531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-envoy\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.427003 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.426985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-podinfo\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.434910 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.434887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-token\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.435342 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.435319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvllg\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-kube-api-access-qvllg\") pod \"e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") " pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.538653 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.538563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:10.719638 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.719608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"]
Jul 24 13:31:10.721324 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:31:10.721294    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c0f8d6_e595_4446_81be_b5891c8aae79.slice/crio-38246b23684a2af69b34b037b29c2b1c481dc13ebaf8c5c28001ae6feefd913f WatchSource:0}: Error finding container 38246b23684a2af69b34b037b29c2b1c481dc13ebaf8c5c28001ae6feefd913f: Status 404 returned error can't find the container with id 38246b23684a2af69b34b037b29c2b1c481dc13ebaf8c5c28001ae6feefd913f
Jul 24 13:31:10.723577 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.723548    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:10.723650 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.723611    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:10.723650 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.723639    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:10.808449 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.807435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" event={"ID":"97c0f8d6-e595-4446-81be-b5891c8aae79","Type":"ContainerStarted","Data":"f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a"}
Jul 24 13:31:10.808449 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.807496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" event={"ID":"97c0f8d6-e595-4446-81be-b5891c8aae79","Type":"ContainerStarted","Data":"38246b23684a2af69b34b037b29c2b1c481dc13ebaf8c5c28001ae6feefd913f"}
Jul 24 13:31:10.834782 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:10.834722    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" podStartSLOduration=0.834703374 podStartE2EDuration="834.703374ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:10.830548072 +0000 UTC m=+2294.819866761" watchObservedRunningTime="2026-07-24 13:31:10.834703374 +0000 UTC m=+2294.824022071"
Jul 24 13:31:11.539589 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:11.539558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:11.544500 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:11.544479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:11.818248 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:11.818148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:11.819891 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:11.819870    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:12.781082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:12.781057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:13.003100 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.001287    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"]
Jul 24 13:31:13.003100 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.001989    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00fa5a8d-5c0d-4b90-ba1b-2c8e1f18b293" containerName="istio-proxy"
Jul 24 13:31:13.014057 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.014024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"]
Jul 24 13:31:13.014240 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.014191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.017950 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.017755    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a9463a-serving-cert\""
Jul 24 13:31:13.050510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.050408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.050510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.050444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m82th\" (UniqueName: \"kubernetes.io/projected/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-kube-api-access-m82th\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.050752 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.050605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.151478 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.151439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.151621 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.151558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.151621 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.151577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m82th\" (UniqueName: \"kubernetes.io/projected/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-kube-api-access-m82th\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.151621 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:31:13.151586    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-serving-cert: secret "maas-api-e2e-api-b-a9463a-serving-cert" not found
Jul 24 13:31:13.151814 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:31:13.151644    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls podName:fd3d1cc1-809a-4754-b6a7-6bdfb8de6438 nodeName:}" failed. No retries permitted until 2026-07-24 13:31:13.651629081 +0000 UTC m=+2297.640947755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls") pod "maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4" (UID: "fd3d1cc1-809a-4754-b6a7-6bdfb8de6438") : secret "maas-api-e2e-api-b-a9463a-serving-cert" not found
Jul 24 13:31:13.152254 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.152233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.166371 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.166344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m82th\" (UniqueName: \"kubernetes.io/projected/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-kube-api-access-m82th\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.656652 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.656613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.659342 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.659311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls\") pod \"maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:13.926550 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:13.926464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:14.095720 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:14.095689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"]
Jul 24 13:31:14.098155 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:31:14.098125    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3d1cc1_809a_4754_b6a7_6bdfb8de6438.slice/crio-09ddcf26893a036545aff92b53cc85a9c682f6689387bf0e85b626866056a9d4 WatchSource:0}: Error finding container 09ddcf26893a036545aff92b53cc85a9c682f6689387bf0e85b626866056a9d4: Status 404 returned error can't find the container with id 09ddcf26893a036545aff92b53cc85a9c682f6689387bf0e85b626866056a9d4
Jul 24 13:31:14.838371 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:14.838276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4" event={"ID":"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438","Type":"ContainerStarted","Data":"6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790"}
Jul 24 13:31:14.838371 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:14.838342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4" event={"ID":"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438","Type":"ContainerStarted","Data":"09ddcf26893a036545aff92b53cc85a9c682f6689387bf0e85b626866056a9d4"}
Jul 24 13:31:14.838591 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:14.838385    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:14.859394 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:14.859345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4" podStartSLOduration=2.41070658 podStartE2EDuration="2.859330939s" totalImagesPullingTime="448.624359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:14.856964997 +0000 UTC m=+2298.846283694" watchObservedRunningTime="2026-07-24 13:31:14.859330939 +0000 UTC m=+2298.848649637"
Jul 24 13:31:15.844831 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:15.844785    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c537101-73f8-4092-9e96-bb1046826563" containerID="5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41" exitCode=0
Jul 24 13:31:15.845345 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:15.844851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" event={"ID":"2c537101-73f8-4092-9e96-bb1046826563","Type":"ContainerDied","Data":"5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41"}
Jul 24 13:31:15.845345 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:15.845335    2579 scope.go:117] "RemoveContainer" containerID="5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41"
Jul 24 13:31:15.846416 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:15.846381    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:15.846510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:15.846442    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:15.846510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:15.846474    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:16.852485 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:16.852453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" event={"ID":"2c537101-73f8-4092-9e96-bb1046826563","Type":"ContainerStarted","Data":"1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3"}
Jul 24 13:31:17.144481 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:17.144399    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:17.145465 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:17.145441    2579 patch_prober.go:28] interesting pod/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Jul 24 13:31:17.145583 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:17.145492    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Jul 24 13:31:18.145219 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:18.145189    2579 patch_prober.go:28] interesting pod/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Jul 24 13:31:18.145601 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:18.145238    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Jul 24 13:31:18.749155 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:18.749121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:19.144767 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:19.144673    2579 patch_prober.go:28] interesting pod/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Jul 24 13:31:19.144767 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:19.144738    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Jul 24 13:31:20.145229 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:20.145196    2579 patch_prober.go:28] interesting pod/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Jul 24 13:31:20.145761 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:20.145734    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Jul 24 13:31:20.851253 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:20.851221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:31:21.149308 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:21.149222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:21.150356 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:21.150336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:25.908805 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:25.908770    2579 generic.go:356] "Generic (PLEG): container finished" podID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerID="f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a" exitCode=0
Jul 24 13:31:25.909233 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:25.908845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" event={"ID":"97c0f8d6-e595-4446-81be-b5891c8aae79","Type":"ContainerDied","Data":"f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a"}
Jul 24 13:31:25.909286 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:25.909274    2579 scope.go:117] "RemoveContainer" containerID="f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a"
Jul 24 13:31:25.910281 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:25.910252    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:25.910378 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:25.910318    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:25.910378 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:25.910357    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:31:26.818620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:26.818584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:26.919680 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:26.919635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" event={"ID":"97c0f8d6-e595-4446-81be-b5891c8aae79","Type":"ContainerStarted","Data":"e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864"}
Jul 24 13:31:27.539231 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:27.539187    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:27.540365 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:27.540338    2579 patch_prober.go:28] interesting pod/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 24 13:31:27.540472 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:27.540389    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 24 13:31:28.539860 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:28.539827    2579 patch_prober.go:28] interesting pod/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 24 13:31:28.540264 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:28.539884    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 24 13:31:29.539558 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:29.539524    2579 patch_prober.go:28] interesting pod/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 24 13:31:29.539720 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:29.539583    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 24 13:31:30.539171 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:30.539139    2579 patch_prober.go:28] interesting pod/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 24 13:31:30.539582 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:30.539191    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 24 13:31:31.543629 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:31.543597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:31.544059 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:31.543865    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:31.544720 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:31.544701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:31:41.847910 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:41.847877    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"]
Jul 24 13:31:41.848442 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:41.848136    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj" podUID="2c690e6e-e9db-4788-95b3-82de2252a2de" containerName="maas-api" containerID="cri-o://bce5988f5517e3dc8113342167daa82a8900d68027b274b0750ad187a28e82d3" gracePeriod=30
Jul 24 13:31:42.005529 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.005498    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c690e6e-e9db-4788-95b3-82de2252a2de" containerID="bce5988f5517e3dc8113342167daa82a8900d68027b274b0750ad187a28e82d3" exitCode=0
Jul 24 13:31:42.005688 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.005566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj" event={"ID":"2c690e6e-e9db-4788-95b3-82de2252a2de","Type":"ContainerDied","Data":"bce5988f5517e3dc8113342167daa82a8900d68027b274b0750ad187a28e82d3"}
Jul 24 13:31:42.101190 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.101112    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:42.233533 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.233504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c690e6e-e9db-4788-95b3-82de2252a2de-openshift-service-ca\") pod \"2c690e6e-e9db-4788-95b3-82de2252a2de\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") "
Jul 24 13:31:42.233694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.233555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxlq5\" (UniqueName: \"kubernetes.io/projected/2c690e6e-e9db-4788-95b3-82de2252a2de-kube-api-access-lxlq5\") pod \"2c690e6e-e9db-4788-95b3-82de2252a2de\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") "
Jul 24 13:31:42.233694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.233609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c690e6e-e9db-4788-95b3-82de2252a2de-maas-api-tls\") pod \"2c690e6e-e9db-4788-95b3-82de2252a2de\" (UID: \"2c690e6e-e9db-4788-95b3-82de2252a2de\") "
Jul 24 13:31:42.233873 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.233851    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c690e6e-e9db-4788-95b3-82de2252a2de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c690e6e-e9db-4788-95b3-82de2252a2de" (UID: "2c690e6e-e9db-4788-95b3-82de2252a2de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:31:42.235817 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.235790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c690e6e-e9db-4788-95b3-82de2252a2de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c690e6e-e9db-4788-95b3-82de2252a2de" (UID: "2c690e6e-e9db-4788-95b3-82de2252a2de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:31:42.235817 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.235804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c690e6e-e9db-4788-95b3-82de2252a2de-kube-api-access-lxlq5" (OuterVolumeSpecName: "kube-api-access-lxlq5") pod "2c690e6e-e9db-4788-95b3-82de2252a2de" (UID: "2c690e6e-e9db-4788-95b3-82de2252a2de"). InnerVolumeSpecName "kube-api-access-lxlq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:31:42.334383 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.334361    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlq5\" (UniqueName: \"kubernetes.io/projected/2c690e6e-e9db-4788-95b3-82de2252a2de-kube-api-access-lxlq5\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:42.334383 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.334382    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c690e6e-e9db-4788-95b3-82de2252a2de-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:42.334522 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:42.334393    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c690e6e-e9db-4788-95b3-82de2252a2de-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:43.011942 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:43.011892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj" event={"ID":"2c690e6e-e9db-4788-95b3-82de2252a2de","Type":"ContainerDied","Data":"2a7a743e53f1382c681fd83479a381bf34e83223c69ec90bdad0f203bfff4bc2"}
Jul 24 13:31:43.012358 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:43.011944    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"
Jul 24 13:31:43.012358 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:43.011979    2579 scope.go:117] "RemoveContainer" containerID="bce5988f5517e3dc8113342167daa82a8900d68027b274b0750ad187a28e82d3"
Jul 24 13:31:43.032738 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:43.032711    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"]
Jul 24 13:31:43.037473 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:43.037452    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e530f2-f9bd6cb4d-4wvrj"]
Jul 24 13:31:44.562788 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:44.562755    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c690e6e-e9db-4788-95b3-82de2252a2de" path="/var/lib/kubelet/pods/2c690e6e-e9db-4788-95b3-82de2252a2de/volumes"
Jul 24 13:31:52.769416 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:52.769336    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"]
Jul 24 13:31:52.769811 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:52.769525    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy" containerID="cri-o://1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3" gracePeriod=30
Jul 24 13:31:58.031121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.031068    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:58.086219 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c537101-73f8-4092-9e96-bb1046826563-istiod-ca-cert\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086250    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-socket\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-credential-socket\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-istio-token\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086534 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztg6f\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-kube-api-access-ztg6f\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086534 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-certs\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086534 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086449    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-data\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086534 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c537101-73f8-4092-9e96-bb1046826563-istio-podinfo\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086534 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:58.086790 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-envoy\") pod \"2c537101-73f8-4092-9e96-bb1046826563\" (UID: \"2c537101-73f8-4092-9e96-bb1046826563\") "
Jul 24 13:31:58.086790 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:58.086790 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:58.086790 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.086756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c537101-73f8-4092-9e96-bb1046826563-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:31:58.087082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.087029    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.087082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.087049    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c537101-73f8-4092-9e96-bb1046826563-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.087082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.087062    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.087082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.087075    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.087540 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.087516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:58.089528 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.089496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:31:58.089677 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.089566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-kube-api-access-ztg6f" (OuterVolumeSpecName: "kube-api-access-ztg6f") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "kube-api-access-ztg6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:31:58.089677 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.089592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:31:58.089886 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.089865    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c537101-73f8-4092-9e96-bb1046826563-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c537101-73f8-4092-9e96-bb1046826563" (UID: "2c537101-73f8-4092-9e96-bb1046826563"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:31:58.094903 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.094877    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c537101-73f8-4092-9e96-bb1046826563" containerID="1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3" exitCode=0
Jul 24 13:31:58.095025 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.094947    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"
Jul 24 13:31:58.095025 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.095006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" event={"ID":"2c537101-73f8-4092-9e96-bb1046826563","Type":"ContainerDied","Data":"1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3"}
Jul 24 13:31:58.095145 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.095057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn" event={"ID":"2c537101-73f8-4092-9e96-bb1046826563","Type":"ContainerDied","Data":"05097041075e384c74e84d42bdbfd83821c16502af2c7a2666ece73a7992757e"}
Jul 24 13:31:58.095145 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.095081    2579 scope.go:117] "RemoveContainer" containerID="1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3"
Jul 24 13:31:58.110480 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.110463    2579 scope.go:117] "RemoveContainer" containerID="5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41"
Jul 24 13:31:58.118608 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.118591    2579 scope.go:117] "RemoveContainer" containerID="1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3"
Jul 24 13:31:58.118878 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:31:58.118860    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3\": container with ID starting with 1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3 not found: ID does not exist" containerID="1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3"
Jul 24 13:31:58.119016 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.118885    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3"} err="failed to get container status \"1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3\": rpc error: code = NotFound desc = could not find container \"1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3\": container with ID starting with 1ed3fde60acdf7ba3d2672d8f0f0750cc779fa0583851fefde8ef99532a119c3 not found: ID does not exist"
Jul 24 13:31:58.119016 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.118903    2579 scope.go:117] "RemoveContainer" containerID="5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41"
Jul 24 13:31:58.119164 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:31:58.119145    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41\": container with ID starting with 5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41 not found: ID does not exist" containerID="5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41"
Jul 24 13:31:58.119202 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.119170    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41"} err="failed to get container status \"5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41\": rpc error: code = NotFound desc = could not find container \"5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41\": container with ID starting with 5427b3b9547ee6ba3443160111216276104cfc773e1135f534b98a828d0d5c41 not found: ID does not exist"
Jul 24 13:31:58.127640 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.127607    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"]
Jul 24 13:31:58.130657 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.130637    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e530f2-openshift-default-7b5b5c6b58-2b9tn"]
Jul 24 13:31:58.187601 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.187577    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.187601 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.187598    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztg6f\" (UniqueName: \"kubernetes.io/projected/2c537101-73f8-4092-9e96-bb1046826563-kube-api-access-ztg6f\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.187764 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.187607    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.187764 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.187615    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c537101-73f8-4092-9e96-bb1046826563-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.187764 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.187623    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c537101-73f8-4092-9e96-bb1046826563-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:31:58.568057 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:31:58.568011    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c537101-73f8-4092-9e96-bb1046826563" path="/var/lib/kubelet/pods/2c537101-73f8-4092-9e96-bb1046826563/volumes"
Jul 24 13:32:08.666458 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.666422    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"]
Jul 24 13:32:08.666898 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.666622    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4" podUID="fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" containerName="maas-api" containerID="cri-o://6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790" gracePeriod=30
Jul 24 13:32:08.914892 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.914864    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:32:08.986949 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.986843    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls\") pod \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") "
Jul 24 13:32:08.987104 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.986966    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-openshift-service-ca\") pod \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") "
Jul 24 13:32:08.987104 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.987027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m82th\" (UniqueName: \"kubernetes.io/projected/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-kube-api-access-m82th\") pod \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\" (UID: \"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438\") "
Jul 24 13:32:08.987391 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.987359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" (UID: "fd3d1cc1-809a-4754-b6a7-6bdfb8de6438"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:32:08.989783 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.989761    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" (UID: "fd3d1cc1-809a-4754-b6a7-6bdfb8de6438"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:32:08.989871 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:08.989786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-kube-api-access-m82th" (OuterVolumeSpecName: "kube-api-access-m82th") pod "fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" (UID: "fd3d1cc1-809a-4754-b6a7-6bdfb8de6438"). InnerVolumeSpecName "kube-api-access-m82th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:32:09.088557 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.088528    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:09.088557 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.088553    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m82th\" (UniqueName: \"kubernetes.io/projected/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-kube-api-access-m82th\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:09.088557 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.088563    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:09.153340 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.153315    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" containerID="6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790" exitCode=0
Jul 24 13:32:09.153498 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.153357    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"
Jul 24 13:32:09.153498 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.153405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4" event={"ID":"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438","Type":"ContainerDied","Data":"6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790"}
Jul 24 13:32:09.153498 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.153452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4" event={"ID":"fd3d1cc1-809a-4754-b6a7-6bdfb8de6438","Type":"ContainerDied","Data":"09ddcf26893a036545aff92b53cc85a9c682f6689387bf0e85b626866056a9d4"}
Jul 24 13:32:09.153498 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.153474    2579 scope.go:117] "RemoveContainer" containerID="6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790"
Jul 24 13:32:09.162642 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.162625    2579 scope.go:117] "RemoveContainer" containerID="6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790"
Jul 24 13:32:09.162887 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:32:09.162870    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790\": container with ID starting with 6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790 not found: ID does not exist" containerID="6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790"
Jul 24 13:32:09.162957 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.162898    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790"} err="failed to get container status \"6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790\": rpc error: code = NotFound desc = could not find container \"6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790\": container with ID starting with 6909fcdc511d40fcae705c3ff134ece508fc19a92a3993e3df40342a14f41790 not found: ID does not exist"
Jul 24 13:32:09.179517 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.179494    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"]
Jul 24 13:32:09.183855 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:09.183834    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9463a-6797cb47c4-lwrz4"]
Jul 24 13:32:10.563106 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:10.563075    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" path="/var/lib/kubelet/pods/fd3d1cc1-809a-4754-b6a7-6bdfb8de6438/volumes"
Jul 24 13:32:21.294155 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:21.294121    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"]
Jul 24 13:32:21.294638 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:21.294409    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy" containerID="cri-o://e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864" gracePeriod=30
Jul 24 13:32:22.277213 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277178    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"]
Jul 24 13:32:22.277613 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277596    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" containerName="maas-api"
Jul 24 13:32:22.277691 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277616    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" containerName="maas-api"
Jul 24 13:32:22.277691 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277636    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy"
Jul 24 13:32:22.277691 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277645    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy"
Jul 24 13:32:22.277691 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277670    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c690e6e-e9db-4788-95b3-82de2252a2de" containerName="maas-api"
Jul 24 13:32:22.277691 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277679    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c690e6e-e9db-4788-95b3-82de2252a2de" containerName="maas-api"
Jul 24 13:32:22.277967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277774    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c690e6e-e9db-4788-95b3-82de2252a2de" containerName="maas-api"
Jul 24 13:32:22.277967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277789    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy"
Jul 24 13:32:22.277967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277799    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3d1cc1-809a-4754-b6a7-6bdfb8de6438" containerName="maas-api"
Jul 24 13:32:22.277967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.277808    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy"
Jul 24 13:32:22.282311 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.282287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.284972 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.284944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-f3a913-openshift-default-dockercfg-2bqwp\""
Jul 24 13:32:22.293392 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.293366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"]
Jul 24 13:32:22.400138 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-socket\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.400510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-podinfo\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.400510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-certs\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.400510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-token\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.400510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-envoy\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.400510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-data\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.400510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhvq\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-kube-api-access-swhvq\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.400510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istiod-ca-cert\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.400510 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.400362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-credential-socket\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501148 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-certs\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501148 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-token\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501391 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-envoy\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501391 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-data\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501391 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhvq\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-kube-api-access-swhvq\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501538 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istiod-ca-cert\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501538 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-credential-socket\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501538 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-certs\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501698 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-socket\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501698 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-podinfo\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501794 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-data\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501905 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-socket\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.501905 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.501894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-credential-socket\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.502170 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.502147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istiod-ca-cert\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.503564 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.503543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-envoy\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.504112 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.504089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-podinfo\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.511912 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.511887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-token\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.512110 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.512082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhvq\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-kube-api-access-swhvq\") pod \"e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") " pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.593758 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.593675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:22.752308 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.752279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"]
Jul 24 13:32:22.752308 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:32:22.752291    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8b9e3a_c8a6_482a_9592_9344ace6ffc5.slice/crio-a1e66e2493010e708487057bf6f62767badb8f90f8153d3e3f11392e9f758503 WatchSource:0}: Error finding container a1e66e2493010e708487057bf6f62767badb8f90f8153d3e3f11392e9f758503: Status 404 returned error can't find the container with id a1e66e2493010e708487057bf6f62767badb8f90f8153d3e3f11392e9f758503
Jul 24 13:32:22.754498 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.754463    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:22.754607 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.754540    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:22.754607 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:22.754582    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:23.229059 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:23.229019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" event={"ID":"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5","Type":"ContainerStarted","Data":"e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106"}
Jul 24 13:32:23.229266 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:23.229112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" event={"ID":"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5","Type":"ContainerStarted","Data":"a1e66e2493010e708487057bf6f62767badb8f90f8153d3e3f11392e9f758503"}
Jul 24 13:32:23.254496 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:23.254443    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" podStartSLOduration=1.254427994 podStartE2EDuration="1.254427994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:23.25280722 +0000 UTC m=+2367.242125908" watchObservedRunningTime="2026-07-24 13:32:23.254427994 +0000 UTC m=+2367.243746689"
Jul 24 13:32:23.594846 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:23.594740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:23.599802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:23.599776    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:24.234339 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:24.234303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:24.235370 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:24.235353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:25.373745 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.373711    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"]
Jul 24 13:32:25.374133 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.374091    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy"
Jul 24 13:32:25.374133 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.374103    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c537101-73f8-4092-9e96-bb1046826563" containerName="istio-proxy"
Jul 24 13:32:25.377298 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.377281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:25.380886 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.380861    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-f3a913-serving-cert\""
Jul 24 13:32:25.381024 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.380887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4vkb\""
Jul 24 13:32:25.383853 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.383835    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:32:25.384126 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.384105    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:32:25.397302 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.397270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"]
Jul 24 13:32:25.532592 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.532552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:25.532592 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.532597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:25.532830 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.532718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxr7n\" (UniqueName: \"kubernetes.io/projected/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-kube-api-access-wxr7n\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:25.633962 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.633821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxr7n\" (UniqueName: \"kubernetes.io/projected/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-kube-api-access-wxr7n\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:25.633962 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.633875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:25.633962 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.633907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:25.634249 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:32:25.634057    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-serving-cert: secret "maas-api-e2e-shared-a-f3a913-serving-cert" not found
Jul 24 13:32:25.634249 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:32:25.634154    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls podName:4a8fbe7c-4288-431e-bd01-bdd54bf12d56 nodeName:}" failed. No retries permitted until 2026-07-24 13:32:26.134135723 +0000 UTC m=+2370.123454402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls") pod "maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7" (UID: "4a8fbe7c-4288-431e-bd01-bdd54bf12d56") : secret "maas-api-e2e-shared-a-f3a913-serving-cert" not found
Jul 24 13:32:25.634538 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.634521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:25.647154 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:25.647118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxr7n\" (UniqueName: \"kubernetes.io/projected/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-kube-api-access-wxr7n\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:26.139911 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.139876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:26.142766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.142738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls\") pod \"maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:26.288004 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.287959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:26.435084 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.433312    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"]
Jul 24 13:32:26.437434 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:32:26.437384    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8fbe7c_4288_431e_bd01_bdd54bf12d56.slice/crio-6b308e8700b38e424af5bd5ebf1db722551eefc158aa72e43c0e97ac87fde7aa WatchSource:0}: Error finding container 6b308e8700b38e424af5bd5ebf1db722551eefc158aa72e43c0e97ac87fde7aa: Status 404 returned error can't find the container with id 6b308e8700b38e424af5bd5ebf1db722551eefc158aa72e43c0e97ac87fde7aa
Jul 24 13:32:26.571003 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.570977    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:32:26.649538 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649502    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-token\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.649695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649562    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-envoy\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.649695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-data\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.649695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c0f8d6-e595-4446-81be-b5891c8aae79-istiod-ca-cert\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.649863 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649702    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-podinfo\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.649863 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-certs\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.649863 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649798    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-credential-socket\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.649863 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649848    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvllg\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-kube-api-access-qvllg\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.650113 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.649877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-socket\") pod \"97c0f8d6-e595-4446-81be-b5891c8aae79\" (UID: \"97c0f8d6-e595-4446-81be-b5891c8aae79\") "
Jul 24 13:32:26.650113 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.650064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c0f8d6-e595-4446-81be-b5891c8aae79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:32:26.650217 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.650136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:26.650306 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.650284    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:26.650306 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.650287    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:26.650442 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.650305    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c0f8d6-e595-4446-81be-b5891c8aae79-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:26.650442 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.650305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:26.650442 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.650413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-data" (OuterVolumeSpecName: "istio-data") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:26.651980 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.651957    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:26.652118 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.652099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-token" (OuterVolumeSpecName: "istio-token") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:32:26.652182 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.652110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-kube-api-access-qvllg" (OuterVolumeSpecName: "kube-api-access-qvllg") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "kube-api-access-qvllg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:32:26.652470 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.652455    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97c0f8d6-e595-4446-81be-b5891c8aae79" (UID: "97c0f8d6-e595-4446-81be-b5891c8aae79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:32:26.751348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.751278    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:26.751348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.751303    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:26.751348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.751311    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:26.751348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.751319    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c0f8d6-e595-4446-81be-b5891c8aae79-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:26.751348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.751327    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:26.751348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.751336    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvllg\" (UniqueName: \"kubernetes.io/projected/97c0f8d6-e595-4446-81be-b5891c8aae79-kube-api-access-qvllg\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:26.751348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:26.751345    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c0f8d6-e595-4446-81be-b5891c8aae79-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:32:27.253135 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.253098    2579 generic.go:356] "Generic (PLEG): container finished" podID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerID="e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864" exitCode=0
Jul 24 13:32:27.253299 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.253153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"
Jul 24 13:32:27.253299 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.253178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" event={"ID":"97c0f8d6-e595-4446-81be-b5891c8aae79","Type":"ContainerDied","Data":"e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864"}
Jul 24 13:32:27.253299 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.253242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd" event={"ID":"97c0f8d6-e595-4446-81be-b5891c8aae79","Type":"ContainerDied","Data":"38246b23684a2af69b34b037b29c2b1c481dc13ebaf8c5c28001ae6feefd913f"}
Jul 24 13:32:27.253299 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.253268    2579 scope.go:117] "RemoveContainer" containerID="e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864"
Jul 24 13:32:27.254954 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.254910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7" event={"ID":"4a8fbe7c-4288-431e-bd01-bdd54bf12d56","Type":"ContainerStarted","Data":"b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a"}
Jul 24 13:32:27.255088 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.254974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7" event={"ID":"4a8fbe7c-4288-431e-bd01-bdd54bf12d56","Type":"ContainerStarted","Data":"6b308e8700b38e424af5bd5ebf1db722551eefc158aa72e43c0e97ac87fde7aa"}
Jul 24 13:32:27.255088 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.255016    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:27.263213 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.263199    2579 scope.go:117] "RemoveContainer" containerID="f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a"
Jul 24 13:32:27.271566 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.271546    2579 scope.go:117] "RemoveContainer" containerID="e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864"
Jul 24 13:32:27.271812 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:32:27.271795    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864\": container with ID starting with e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864 not found: ID does not exist" containerID="e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864"
Jul 24 13:32:27.271864 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.271820    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864"} err="failed to get container status \"e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864\": rpc error: code = NotFound desc = could not find container \"e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864\": container with ID starting with e3473a0f46b77589815f941a69995e1c871dc3fede73e160ddfb66d727a7d864 not found: ID does not exist"
Jul 24 13:32:27.271864 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.271836    2579 scope.go:117] "RemoveContainer" containerID="f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a"
Jul 24 13:32:27.272113 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:32:27.272094    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a\": container with ID starting with f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a not found: ID does not exist" containerID="f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a"
Jul 24 13:32:27.272169 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.272122    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a"} err="failed to get container status \"f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a\": rpc error: code = NotFound desc = could not find container \"f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a\": container with ID starting with f7f7ef832f2430d4866261a8d0a821151063eae521b1c782b2451a2b4bb23e8a not found: ID does not exist"
Jul 24 13:32:27.278748 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.278708    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7" podStartSLOduration=1.718771343 podStartE2EDuration="2.278693627s" totalImagesPullingTime="559.922284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:27.277549965 +0000 UTC m=+2371.266868665" watchObservedRunningTime="2026-07-24 13:32:27.278693627 +0000 UTC m=+2371.268012324"
Jul 24 13:32:27.294806 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.294773    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"]
Jul 24 13:32:27.300660 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:27.300641    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a9463a-openshift-default-787754c7db-vv8qd"]
Jul 24 13:32:28.563556 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:28.563522    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" path="/var/lib/kubelet/pods/97c0f8d6-e595-4446-81be-b5891c8aae79/volumes"
Jul 24 13:32:30.442840 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.442808    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"]
Jul 24 13:32:30.443382 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.443361    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy"
Jul 24 13:32:30.443453 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.443386    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy"
Jul 24 13:32:30.443453 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.443414    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy"
Jul 24 13:32:30.443453 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.443424    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy"
Jul 24 13:32:30.443603 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.443522    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy"
Jul 24 13:32:30.448227 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.448196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.452775 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.452751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b1a3df-openshift-default-dockercfg-mlfw8\""
Jul 24 13:32:30.468679 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.468651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"]
Jul 24 13:32:30.487691 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.487654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-socket\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.487855 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.487735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-envoy\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.487855 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.487787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78d5r\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-kube-api-access-78d5r\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.487976 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.487894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-credential-socket\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.488019 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.487990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-certs\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.488085 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.488053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-podinfo\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.488121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.488086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-data\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.488158 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.488122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istiod-ca-cert\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.488158 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.488149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-token\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589168 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-podinfo\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-data\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istiod-ca-cert\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-token\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589368 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-socket\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589573 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-envoy\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589573 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78d5r\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-kube-api-access-78d5r\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589679 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-credential-socket\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589736 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-data\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.589736 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-certs\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.590014 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.589973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-credential-socket\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.590230 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.590189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-socket\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.590471 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.590444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-certs\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.590700 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.590680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istiod-ca-cert\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.592223 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.592202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-envoy\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.592456 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.592436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-podinfo\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.598484 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.598462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-token\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.598617 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.598600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78d5r\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-kube-api-access-78d5r\") pod \"e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") " pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.764556 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.764466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:30.907345 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.907318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"]
Jul 24 13:32:30.908350 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:32:30.908328    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c7e77a_3ae1_497c_98de_62f2e5b4bda8.slice/crio-ff23d416f63e39011495d36df199848751d50798c4116a3d9bc746f0eca49f87 WatchSource:0}: Error finding container ff23d416f63e39011495d36df199848751d50798c4116a3d9bc746f0eca49f87: Status 404 returned error can't find the container with id ff23d416f63e39011495d36df199848751d50798c4116a3d9bc746f0eca49f87
Jul 24 13:32:30.910640 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.910590    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:30.910746 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.910672    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:30.910746 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:30.910713    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:31.279248 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:31.279214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" event={"ID":"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8","Type":"ContainerStarted","Data":"707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1"}
Jul 24 13:32:31.279428 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:31.279273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" event={"ID":"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8","Type":"ContainerStarted","Data":"ff23d416f63e39011495d36df199848751d50798c4116a3d9bc746f0eca49f87"}
Jul 24 13:32:31.305311 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:31.305264    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" podStartSLOduration=1.305248095 podStartE2EDuration="1.305248095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:31.302700142 +0000 UTC m=+2375.292018836" watchObservedRunningTime="2026-07-24 13:32:31.305248095 +0000 UTC m=+2375.294566789"
Jul 24 13:32:31.765380 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:31.765335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:31.770238 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:31.770212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:32.285506 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:32.285478    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:32.286413 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:32.286393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:33.268688 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.268649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:32:33.470914 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.470871    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"]
Jul 24 13:32:33.471562 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.471544    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c0f8d6-e595-4446-81be-b5891c8aae79" containerName="istio-proxy"
Jul 24 13:32:33.475184 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.475160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:33.477992 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.477972    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b1a3df-serving-cert\""
Jul 24 13:32:33.490573 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.490545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"]
Jul 24 13:32:33.519876 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.519794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:33.519876 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.519831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlddb\" (UniqueName: \"kubernetes.io/projected/6c27e23f-fa01-402e-b276-d0a9eb35d967-kube-api-access-dlddb\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:33.520085 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.519971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c27e23f-fa01-402e-b276-d0a9eb35d967-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:33.621290 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.621252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c27e23f-fa01-402e-b276-d0a9eb35d967-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:33.621470 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.621310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:33.621470 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.621330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlddb\" (UniqueName: \"kubernetes.io/projected/6c27e23f-fa01-402e-b276-d0a9eb35d967-kube-api-access-dlddb\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:33.621470 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:32:33.621423    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-serving-cert: secret "maas-api-e2e-shared-b-b1a3df-serving-cert" not found
Jul 24 13:32:33.621657 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:32:33.621488    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls podName:6c27e23f-fa01-402e-b276-d0a9eb35d967 nodeName:}" failed. No retries permitted until 2026-07-24 13:32:34.121471917 +0000 UTC m=+2378.110790591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls") pod "maas-api-e2e-shared-b-b1a3df-54fb58754-569jl" (UID: "6c27e23f-fa01-402e-b276-d0a9eb35d967") : secret "maas-api-e2e-shared-b-b1a3df-serving-cert" not found
Jul 24 13:32:33.622058 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.622037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c27e23f-fa01-402e-b276-d0a9eb35d967-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:33.633020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:33.632998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlddb\" (UniqueName: \"kubernetes.io/projected/6c27e23f-fa01-402e-b276-d0a9eb35d967-kube-api-access-dlddb\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:34.125791 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:34.125746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:34.128498 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:34.128468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1a3df-54fb58754-569jl\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:34.387749 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:34.387656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:34.562767 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:32:34.562735    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c27e23f_fa01_402e_b276_d0a9eb35d967.slice/crio-14d725d1a0d2539a63779614fb78f6e94cc9a8eac318bf796f413bbd8ec1dfc8 WatchSource:0}: Error finding container 14d725d1a0d2539a63779614fb78f6e94cc9a8eac318bf796f413bbd8ec1dfc8: Status 404 returned error can't find the container with id 14d725d1a0d2539a63779614fb78f6e94cc9a8eac318bf796f413bbd8ec1dfc8
Jul 24 13:32:34.564199 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:34.564175    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"]
Jul 24 13:32:35.311695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:35.311586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl" event={"ID":"6c27e23f-fa01-402e-b276-d0a9eb35d967","Type":"ContainerStarted","Data":"46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10"}
Jul 24 13:32:35.311695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:35.311660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl" event={"ID":"6c27e23f-fa01-402e-b276-d0a9eb35d967","Type":"ContainerStarted","Data":"14d725d1a0d2539a63779614fb78f6e94cc9a8eac318bf796f413bbd8ec1dfc8"}
Jul 24 13:32:35.312127 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:35.312106    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:35.338875 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:35.338813    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl" podStartSLOduration=1.927568048 podStartE2EDuration="2.338795039s" totalImagesPullingTime="411.226991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:35.336211739 +0000 UTC m=+2379.325530433" watchObservedRunningTime="2026-07-24 13:32:35.338795039 +0000 UTC m=+2379.328113734"
Jul 24 13:32:40.235780 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:40.235734    2579 patch_prober.go:28] interesting pod/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 24 13:32:40.236372 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:40.235819    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 13:32:41.325080 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:41.325049    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:32:42.355082 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:42.355048    2579 generic.go:356] "Generic (PLEG): container finished" podID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerID="e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106" exitCode=0
Jul 24 13:32:42.355489 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:42.355124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" event={"ID":"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5","Type":"ContainerDied","Data":"e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106"}
Jul 24 13:32:42.355543 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:42.355530    2579 scope.go:117] "RemoveContainer" containerID="e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106"
Jul 24 13:32:42.356565 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:42.356541    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:42.356647 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:42.356618    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:42.356704 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:42.356659    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:43.362689 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:43.362652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" event={"ID":"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5","Type":"ContainerStarted","Data":"453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82"}
Jul 24 13:32:43.594313 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:43.594272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:43.594529 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:43.594325    2579 patch_prober.go:28] interesting pod/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 13:32:43.594529 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:43.594379    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 13:32:44.594435 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:44.594396    2579 patch_prober.go:28] interesting pod/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 13:32:44.595086 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:44.595050    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 13:32:45.594285 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:45.594240    2579 patch_prober.go:28] interesting pod/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 13:32:45.594497 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:45.594328    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 13:32:46.594161 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:46.594122    2579 patch_prober.go:28] interesting pod/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 13:32:46.594370 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:46.594206    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 13:32:47.598251 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:47.598221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:47.598643 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:47.598604    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:47.599463 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:47.599444    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:32:47.933281 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:47.933247    2579 patch_prober.go:28] interesting pod/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": read tcp 10.133.0.2:59278->10.133.0.79:15021: read: connection reset by peer" start-of-body=
Jul 24 13:32:47.933444 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:47.933330    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": read tcp 10.133.0.2:59278->10.133.0.79:15021: read: connection reset by peer"
Jul 24 13:32:48.392506 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:48.392409    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerID="707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1" exitCode=0
Jul 24 13:32:48.392506 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:48.392494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" event={"ID":"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8","Type":"ContainerDied","Data":"707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1"}
Jul 24 13:32:48.392861 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:48.392846    2579 scope.go:117] "RemoveContainer" containerID="707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1"
Jul 24 13:32:48.393889 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:48.393854    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:48.393998 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:48.393978    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:48.394047 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:48.394028    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:32:49.410967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:49.408357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" event={"ID":"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8","Type":"ContainerStarted","Data":"b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de"}
Jul 24 13:32:49.765513 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:49.765426    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:49.765679 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:49.765636    2579 patch_prober.go:28] interesting pod/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Jul 24 13:32:49.765743 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:49.765685    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Jul 24 13:32:50.765678 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:50.765638    2579 patch_prober.go:28] interesting pod/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Jul 24 13:32:50.766085 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:50.765702    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Jul 24 13:32:51.765828 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:51.765793    2579 patch_prober.go:28] interesting pod/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Jul 24 13:32:51.766271 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:51.765859    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Jul 24 13:32:52.765338 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:52.765298    2579 patch_prober.go:28] interesting pod/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Jul 24 13:32:52.765524 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:52.765370    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Jul 24 13:32:53.769051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:53.769012    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:53.769463 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:53.769299    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:53.770036 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:53.770015    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:32:57.797228 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.797188    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"]
Jul 24 13:32:57.800632 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.800615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:57.803471 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.803452    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f3a913\"/\"kube-root-ca.crt\""
Jul 24 13:32:57.804504 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.804487    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f3a913\"/\"openshift-service-ca.crt\""
Jul 24 13:32:57.804564 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.804502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f3a913\"/\"default-dockercfg-psg89\""
Jul 24 13:32:57.804615 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.804570    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f3a913\"/\"auth-test-model-f3a913-kserve-self-signed-certs\""
Jul 24 13:32:57.812639 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.812609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"]
Jul 24 13:32:57.992672 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.992632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tls-certs\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:57.992861 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.992736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-dshm\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:57.992861 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.992774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tmp-dir\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:57.993013 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.992855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmrj\" (UniqueName: \"kubernetes.io/projected/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-kube-api-access-kcmrj\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:57.993013 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.992903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-model-cache\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:57.993013 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:57.993002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-home\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.093671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.093579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tls-certs\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.093671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.093622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-dshm\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.093671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.093640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tmp-dir\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.094028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.093769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmrj\" (UniqueName: \"kubernetes.io/projected/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-kube-api-access-kcmrj\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.094028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.093822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-model-cache\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.094028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.093887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-home\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.094187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.094044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tmp-dir\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.094226 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.094211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-home\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.094318 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.094304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-model-cache\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.096172 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.096146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-dshm\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.096605 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.096589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tls-certs\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.107684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.107658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmrj\" (UniqueName: \"kubernetes.io/projected/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-kube-api-access-kcmrj\") pod \"auth-test-model-f3a913-kserve-796d4cd7dc-w72fb\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") " pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.110601 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.110578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:32:58.253409 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.253376    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"]
Jul 24 13:32:58.255155 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:32:58.255131    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a18f69d_f9fd_4bc8_99dc_55f0b459a8b2.slice/crio-2d0c6d6d247d40bd9fde787a2407c40c4eecc22c446a027ad66d01e0c775458c WatchSource:0}: Error finding container 2d0c6d6d247d40bd9fde787a2407c40c4eecc22c446a027ad66d01e0c775458c: Status 404 returned error can't find the container with id 2d0c6d6d247d40bd9fde787a2407c40c4eecc22c446a027ad66d01e0c775458c
Jul 24 13:32:58.463462 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.463420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb" event={"ID":"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2","Type":"ContainerStarted","Data":"0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592"}
Jul 24 13:32:58.463462 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.463467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb" event={"ID":"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2","Type":"ContainerStarted","Data":"2d0c6d6d247d40bd9fde787a2407c40c4eecc22c446a027ad66d01e0c775458c"}
Jul 24 13:32:58.483841 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:32:58.483788    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb" podStartSLOduration=1.483771498 podStartE2EDuration="1.483771498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:58.482803442 +0000 UTC m=+2402.472122138" watchObservedRunningTime="2026-07-24 13:32:58.483771498 +0000 UTC m=+2402.473090192"
Jul 24 13:33:08.111292 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:33:08.111248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:33:08.111721 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:33:08.111305    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:33:08.123333 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:33:08.123303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:33:08.538534 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:33:08.538506    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:35:12.478702 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:12.478606    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"]
Jul 24 13:35:27.497738 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.497699    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb" podUID="5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" containerName="main" containerID="cri-o://0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592" gracePeriod=45
Jul 24 13:35:27.744569 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.744545    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:35:27.825632 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.825543    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-model-cache\") pod \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") "
Jul 24 13:35:27.825632 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.825610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-dshm\") pod \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") "
Jul 24 13:35:27.825834 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.825663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tmp-dir\") pod \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") "
Jul 24 13:35:27.825834 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.825699    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcmrj\" (UniqueName: \"kubernetes.io/projected/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-kube-api-access-kcmrj\") pod \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") "
Jul 24 13:35:27.825834 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.825719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-home\") pod \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") "
Jul 24 13:35:27.825834 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.825792    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tls-certs\") pod \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\" (UID: \"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2\") "
Jul 24 13:35:27.826081 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.825866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" (UID: "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:35:27.826081 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.825889    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" (UID: "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:35:27.826187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.826104    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-home" (OuterVolumeSpecName: "home") pod "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" (UID: "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:35:27.826187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.826117    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tmp-dir\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:35:27.826187 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.826157    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-model-cache\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:35:27.828016 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.827980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-dshm" (OuterVolumeSpecName: "dshm") pod "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" (UID: "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:35:27.828016 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.827989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" (UID: "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:35:27.828016 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.827997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-kube-api-access-kcmrj" (OuterVolumeSpecName: "kube-api-access-kcmrj") pod "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" (UID: "5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2"). InnerVolumeSpecName "kube-api-access-kcmrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:35:27.927152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.927121    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcmrj\" (UniqueName: \"kubernetes.io/projected/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-kube-api-access-kcmrj\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:35:27.927152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.927146    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-home\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:35:27.927152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.927154    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-tls-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:35:27.927152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:27.927163    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2-dshm\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:35:28.300497 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.300458    2579 generic.go:356] "Generic (PLEG): container finished" podID="5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" containerID="0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592" exitCode=0
Jul 24 13:35:28.300694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.300504    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"
Jul 24 13:35:28.300694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.300501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb" event={"ID":"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2","Type":"ContainerDied","Data":"0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592"}
Jul 24 13:35:28.300694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.300615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb" event={"ID":"5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2","Type":"ContainerDied","Data":"2d0c6d6d247d40bd9fde787a2407c40c4eecc22c446a027ad66d01e0c775458c"}
Jul 24 13:35:28.300694 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.300639    2579 scope.go:117] "RemoveContainer" containerID="0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592"
Jul 24 13:35:28.313105 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.313090    2579 scope.go:117] "RemoveContainer" containerID="0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592"
Jul 24 13:35:28.313358 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:35:28.313339    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592\": container with ID starting with 0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592 not found: ID does not exist" containerID="0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592"
Jul 24 13:35:28.313422 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.313372    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592"} err="failed to get container status \"0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592\": rpc error: code = NotFound desc = could not find container \"0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592\": container with ID starting with 0aa8df85e3e76fc9f7ea415460dce6ea31cb8e5745cc276fe1705e1e2c427592 not found: ID does not exist"
Jul 24 13:35:28.327671 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.327647    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"]
Jul 24 13:35:28.338954 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.338775    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/auth-test-model-f3a913-kserve-796d4cd7dc-w72fb"]
Jul 24 13:35:28.563020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:28.562944    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" path="/var/lib/kubelet/pods/5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2/volumes"
Jul 24 13:35:43.017026 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.016990    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"]
Jul 24 13:35:43.017431 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.017375    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" containerName="main"
Jul 24 13:35:43.017431 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.017387    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" containerName="main"
Jul 24 13:35:43.017514 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.017450    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a18f69d-f9fd-4bc8-99dc-55f0b459a8b2" containerName="main"
Jul 24 13:35:43.019448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.019432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.022939 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.022902    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"kube-root-ca.crt\""
Jul 24 13:35:43.023053 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.022976    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"default-dockercfg-dk7vn\""
Jul 24 13:35:43.023097 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.022976    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"openshift-service-ca.crt\""
Jul 24 13:35:43.023097 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.022896    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"sub-test-model-b1a3df-kserve-self-signed-certs\""
Jul 24 13:35:43.029447 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.029426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"]
Jul 24 13:35:43.062905 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.062867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rltjj\" (UniqueName: \"kubernetes.io/projected/60f64844-2256-4415-921c-81962341c593-kube-api-access-rltjj\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.062905 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.062909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-home\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.063176 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.062955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-model-cache\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.063176 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.063058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60f64844-2256-4415-921c-81962341c593-tls-certs\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.063176 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.063102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-tmp-dir\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.063176 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.063154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-dshm\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.164268 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.164229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60f64844-2256-4415-921c-81962341c593-tls-certs\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.164472 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.164374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-tmp-dir\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.164558 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.164515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-dshm\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.164621 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.164596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rltjj\" (UniqueName: \"kubernetes.io/projected/60f64844-2256-4415-921c-81962341c593-kube-api-access-rltjj\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.164675 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.164652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-home\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.164732 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.164695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-model-cache\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.165080 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.165015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-tmp-dir\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.165198 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.165107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-model-cache\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.165437 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.165391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-home\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.167335 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.167282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-dshm\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.168095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.168077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60f64844-2256-4415-921c-81962341c593-tls-certs\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.178525 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.178505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rltjj\" (UniqueName: \"kubernetes.io/projected/60f64844-2256-4415-921c-81962341c593-kube-api-access-rltjj\") pod \"sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9\" (UID: \"60f64844-2256-4415-921c-81962341c593\") " pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.329718 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.329618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:43.481106 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:43.481079    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"]
Jul 24 13:35:43.483094 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:35:43.483067    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60f64844_2256_4415_921c_81962341c593.slice/crio-0c6b684f257b3d1f03a6c8fbf38bed6365ec026fcd9b6ed2758abbf6c55c6c44 WatchSource:0}: Error finding container 0c6b684f257b3d1f03a6c8fbf38bed6365ec026fcd9b6ed2758abbf6c55c6c44: Status 404 returned error can't find the container with id 0c6b684f257b3d1f03a6c8fbf38bed6365ec026fcd9b6ed2758abbf6c55c6c44
Jul 24 13:35:44.403207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:44.403170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9" event={"ID":"60f64844-2256-4415-921c-81962341c593","Type":"ContainerStarted","Data":"dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d"}
Jul 24 13:35:44.403207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:44.403214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9" event={"ID":"60f64844-2256-4415-921c-81962341c593","Type":"ContainerStarted","Data":"0c6b684f257b3d1f03a6c8fbf38bed6365ec026fcd9b6ed2758abbf6c55c6c44"}
Jul 24 13:35:44.427836 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:44.427782    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9" podStartSLOduration=2.427766893 podStartE2EDuration="2.427766893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:44.426729707 +0000 UTC m=+2568.416048399" watchObservedRunningTime="2026-07-24 13:35:44.427766893 +0000 UTC m=+2568.417085589"
Jul 24 13:35:53.329979 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:53.329939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:53.330460 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:53.330001    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:53.342166 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:53.342140    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:35:53.464682 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:35:53.464650    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:36:19.855202 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:19.855127    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"]
Jul 24 13:36:34.874172 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:34.874125    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9" podUID="60f64844-2256-4415-921c-81962341c593" containerName="main" containerID="cri-o://dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d" gracePeriod=45
Jul 24 13:36:35.123713 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.123689    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:36:35.147342 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-tmp-dir\") pod \"60f64844-2256-4415-921c-81962341c593\" (UID: \"60f64844-2256-4415-921c-81962341c593\") "
Jul 24 13:36:35.147342 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rltjj\" (UniqueName: \"kubernetes.io/projected/60f64844-2256-4415-921c-81962341c593-kube-api-access-rltjj\") pod \"60f64844-2256-4415-921c-81962341c593\" (UID: \"60f64844-2256-4415-921c-81962341c593\") "
Jul 24 13:36:35.147554 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147356    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60f64844-2256-4415-921c-81962341c593-tls-certs\") pod \"60f64844-2256-4415-921c-81962341c593\" (UID: \"60f64844-2256-4415-921c-81962341c593\") "
Jul 24 13:36:35.147554 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-model-cache\") pod \"60f64844-2256-4415-921c-81962341c593\" (UID: \"60f64844-2256-4415-921c-81962341c593\") "
Jul 24 13:36:35.147554 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-home\") pod \"60f64844-2256-4415-921c-81962341c593\" (UID: \"60f64844-2256-4415-921c-81962341c593\") "
Jul 24 13:36:35.147554 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-dshm\") pod \"60f64844-2256-4415-921c-81962341c593\" (UID: \"60f64844-2256-4415-921c-81962341c593\") "
Jul 24 13:36:35.147766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147572    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60f64844-2256-4415-921c-81962341c593" (UID: "60f64844-2256-4415-921c-81962341c593"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:36:35.147766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-model-cache" (OuterVolumeSpecName: "model-cache") pod "60f64844-2256-4415-921c-81962341c593" (UID: "60f64844-2256-4415-921c-81962341c593"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:36:35.147766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147714    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-tmp-dir\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:36:35.147961 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.147941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-home" (OuterVolumeSpecName: "home") pod "60f64844-2256-4415-921c-81962341c593" (UID: "60f64844-2256-4415-921c-81962341c593"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:36:35.149867 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.149835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f64844-2256-4415-921c-81962341c593-kube-api-access-rltjj" (OuterVolumeSpecName: "kube-api-access-rltjj") pod "60f64844-2256-4415-921c-81962341c593" (UID: "60f64844-2256-4415-921c-81962341c593"). InnerVolumeSpecName "kube-api-access-rltjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:36:35.150132 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.150085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-dshm" (OuterVolumeSpecName: "dshm") pod "60f64844-2256-4415-921c-81962341c593" (UID: "60f64844-2256-4415-921c-81962341c593"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:36:35.150132 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.150105    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f64844-2256-4415-921c-81962341c593-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60f64844-2256-4415-921c-81962341c593" (UID: "60f64844-2256-4415-921c-81962341c593"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:35.248861 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.248812    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rltjj\" (UniqueName: \"kubernetes.io/projected/60f64844-2256-4415-921c-81962341c593-kube-api-access-rltjj\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:36:35.248861 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.248856    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60f64844-2256-4415-921c-81962341c593-tls-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:36:35.248861 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.248871    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-model-cache\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:36:35.249129 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.248883    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-home\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:36:35.249129 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.248894    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60f64844-2256-4415-921c-81962341c593-dshm\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:36:35.684906 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.684870    2579 generic.go:356] "Generic (PLEG): container finished" podID="60f64844-2256-4415-921c-81962341c593" containerID="dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d" exitCode=0
Jul 24 13:36:35.685090 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.684915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9" event={"ID":"60f64844-2256-4415-921c-81962341c593","Type":"ContainerDied","Data":"dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d"}
Jul 24 13:36:35.685090 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.684957    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"
Jul 24 13:36:35.685090 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.684974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9" event={"ID":"60f64844-2256-4415-921c-81962341c593","Type":"ContainerDied","Data":"0c6b684f257b3d1f03a6c8fbf38bed6365ec026fcd9b6ed2758abbf6c55c6c44"}
Jul 24 13:36:35.685090 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.684998    2579 scope.go:117] "RemoveContainer" containerID="dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d"
Jul 24 13:36:35.694957 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.694911    2579 scope.go:117] "RemoveContainer" containerID="dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d"
Jul 24 13:36:35.695238 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:36:35.695211    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d\": container with ID starting with dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d not found: ID does not exist" containerID="dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d"
Jul 24 13:36:35.695359 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.695244    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d"} err="failed to get container status \"dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d\": rpc error: code = NotFound desc = could not find container \"dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d\": container with ID starting with dc9f09590ef72815e7df2de0ccdd55fe4569710e67d009d90f4680a3eb749a3d not found: ID does not exist"
Jul 24 13:36:35.711727 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.711695    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"]
Jul 24 13:36:35.716464 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:35.716436    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/sub-test-model-b1a3df-kserve-8f4ffb45-nt6s9"]
Jul 24 13:36:36.565197 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:36.565164    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f64844-2256-4415-921c-81962341c593" path="/var/lib/kubelet/pods/60f64844-2256-4415-921c-81962341c593/volumes"
Jul 24 13:36:46.811149 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.811109    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"]
Jul 24 13:36:46.811623 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.811607    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f64844-2256-4415-921c-81962341c593" containerName="main"
Jul 24 13:36:46.811623 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.811622    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f64844-2256-4415-921c-81962341c593" containerName="main"
Jul 24 13:36:46.811712 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.811688    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f64844-2256-4415-921c-81962341c593" containerName="main"
Jul 24 13:36:46.814939 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.814894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.819281 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.819145    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"openshift-service-ca.crt\""
Jul 24 13:36:46.819418 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.819339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"default-dockercfg-dk7vn\""
Jul 24 13:36:46.819418 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.819360    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"kube-root-ca.crt\""
Jul 24 13:36:46.819418 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.819345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"rate-test-model-b1a3df-kserve-self-signed-certs\""
Jul 24 13:36:46.825285 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.825265    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"]
Jul 24 13:36:46.852156 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.852135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-model-cache\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.852281 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.852169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4479f-6670-42f7-b4ce-04b7c7a21373-tls-certs\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.852281 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.852190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-tmp-dir\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.852281 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.852256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-home\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.852395 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.852310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvhmq\" (UniqueName: \"kubernetes.io/projected/eed4479f-6670-42f7-b4ce-04b7c7a21373-kube-api-access-pvhmq\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.852395 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.852330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-dshm\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.953714 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.953681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvhmq\" (UniqueName: \"kubernetes.io/projected/eed4479f-6670-42f7-b4ce-04b7c7a21373-kube-api-access-pvhmq\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.953714 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.953716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-dshm\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.953967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.953752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-model-cache\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.953967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.953788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4479f-6670-42f7-b4ce-04b7c7a21373-tls-certs\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.953967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.953818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-tmp-dir\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.953967 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.953858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-home\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.954287 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.954264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-model-cache\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.954361 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.954272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-home\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.954361 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.954311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-tmp-dir\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.956166 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.956148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-dshm\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.956398 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.956382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4479f-6670-42f7-b4ce-04b7c7a21373-tls-certs\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:46.965564 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:46.965540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvhmq\" (UniqueName: \"kubernetes.io/projected/eed4479f-6670-42f7-b4ce-04b7c7a21373-kube-api-access-pvhmq\") pod \"rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") " pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:47.127449 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:47.127363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:47.268450 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:47.268427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"]
Jul 24 13:36:47.270060 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:36:47.270034    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeed4479f_6670_42f7_b4ce_04b7c7a21373.slice/crio-40d5cff76bbf2f989ca557f10688f2025c352fc7cdb85b6adcc30266f78a8aa6 WatchSource:0}: Error finding container 40d5cff76bbf2f989ca557f10688f2025c352fc7cdb85b6adcc30266f78a8aa6: Status 404 returned error can't find the container with id 40d5cff76bbf2f989ca557f10688f2025c352fc7cdb85b6adcc30266f78a8aa6
Jul 24 13:36:47.761809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:47.761776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l" event={"ID":"eed4479f-6670-42f7-b4ce-04b7c7a21373","Type":"ContainerStarted","Data":"5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d"}
Jul 24 13:36:47.761809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:47.761814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l" event={"ID":"eed4479f-6670-42f7-b4ce-04b7c7a21373","Type":"ContainerStarted","Data":"40d5cff76bbf2f989ca557f10688f2025c352fc7cdb85b6adcc30266f78a8aa6"}
Jul 24 13:36:47.784186 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:47.784141    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l" podStartSLOduration=1.784127085 podStartE2EDuration="1.784127085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:36:47.782419008 +0000 UTC m=+2631.771737705" watchObservedRunningTime="2026-07-24 13:36:47.784127085 +0000 UTC m=+2631.773445817"
Jul 24 13:36:57.128564 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:57.128476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:57.128564 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:57.128531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:57.140597 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:57.140575    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:36:57.827807 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:36:57.827774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:37:31.702412 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:31.702380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"]
Jul 24 13:37:46.721134 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:46.721092    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l" podUID="eed4479f-6670-42f7-b4ce-04b7c7a21373" containerName="main" containerID="cri-o://5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d" gracePeriod=45
Jul 24 13:37:46.973900 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:46.973848    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:37:47.085781 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.085748    2579 generic.go:356] "Generic (PLEG): container finished" podID="eed4479f-6670-42f7-b4ce-04b7c7a21373" containerID="5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d" exitCode=0
Jul 24 13:37:47.085991 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.085789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l" event={"ID":"eed4479f-6670-42f7-b4ce-04b7c7a21373","Type":"ContainerDied","Data":"5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d"}
Jul 24 13:37:47.085991 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.085801    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"
Jul 24 13:37:47.085991 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.085815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l" event={"ID":"eed4479f-6670-42f7-b4ce-04b7c7a21373","Type":"ContainerDied","Data":"40d5cff76bbf2f989ca557f10688f2025c352fc7cdb85b6adcc30266f78a8aa6"}
Jul 24 13:37:47.085991 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.085832    2579 scope.go:117] "RemoveContainer" containerID="5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d"
Jul 24 13:37:47.091956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.091918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-dshm\") pod \"eed4479f-6670-42f7-b4ce-04b7c7a21373\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") "
Jul 24 13:37:47.092095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.091969    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4479f-6670-42f7-b4ce-04b7c7a21373-tls-certs\") pod \"eed4479f-6670-42f7-b4ce-04b7c7a21373\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") "
Jul 24 13:37:47.092095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.091989    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvhmq\" (UniqueName: \"kubernetes.io/projected/eed4479f-6670-42f7-b4ce-04b7c7a21373-kube-api-access-pvhmq\") pod \"eed4479f-6670-42f7-b4ce-04b7c7a21373\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") "
Jul 24 13:37:47.092095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.092018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-model-cache\") pod \"eed4479f-6670-42f7-b4ce-04b7c7a21373\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") "
Jul 24 13:37:47.092095 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.092035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-tmp-dir\") pod \"eed4479f-6670-42f7-b4ce-04b7c7a21373\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") "
Jul 24 13:37:47.092365 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.092111    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-home\") pod \"eed4479f-6670-42f7-b4ce-04b7c7a21373\" (UID: \"eed4479f-6670-42f7-b4ce-04b7c7a21373\") "
Jul 24 13:37:47.092365 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.092311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-model-cache" (OuterVolumeSpecName: "model-cache") pod "eed4479f-6670-42f7-b4ce-04b7c7a21373" (UID: "eed4479f-6670-42f7-b4ce-04b7c7a21373"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:37:47.092474 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.092386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eed4479f-6670-42f7-b4ce-04b7c7a21373" (UID: "eed4479f-6670-42f7-b4ce-04b7c7a21373"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:37:47.092547 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.092528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-home" (OuterVolumeSpecName: "home") pod "eed4479f-6670-42f7-b4ce-04b7c7a21373" (UID: "eed4479f-6670-42f7-b4ce-04b7c7a21373"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:37:47.092604 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.092537    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-model-cache\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:37:47.092604 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.092579    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-tmp-dir\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:37:47.094303 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.094276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed4479f-6670-42f7-b4ce-04b7c7a21373-kube-api-access-pvhmq" (OuterVolumeSpecName: "kube-api-access-pvhmq") pod "eed4479f-6670-42f7-b4ce-04b7c7a21373" (UID: "eed4479f-6670-42f7-b4ce-04b7c7a21373"). InnerVolumeSpecName "kube-api-access-pvhmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:37:47.094611 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.094583    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed4479f-6670-42f7-b4ce-04b7c7a21373-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eed4479f-6670-42f7-b4ce-04b7c7a21373" (UID: "eed4479f-6670-42f7-b4ce-04b7c7a21373"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:37:47.094739 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.094723    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-dshm" (OuterVolumeSpecName: "dshm") pod "eed4479f-6670-42f7-b4ce-04b7c7a21373" (UID: "eed4479f-6670-42f7-b4ce-04b7c7a21373"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:37:47.095480 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.095463    2579 scope.go:117] "RemoveContainer" containerID="5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d"
Jul 24 13:37:47.095737 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:37:47.095714    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d\": container with ID starting with 5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d not found: ID does not exist" containerID="5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d"
Jul 24 13:37:47.095845 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.095743    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d"} err="failed to get container status \"5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d\": rpc error: code = NotFound desc = could not find container \"5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d\": container with ID starting with 5dbcc326d60a43be09c82d0f7dd3cabdec54e035e52e5d8024208f0f7dc84d7d not found: ID does not exist"
Jul 24 13:37:47.193480 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.193435    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-dshm\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:37:47.193480 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.193470    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4479f-6670-42f7-b4ce-04b7c7a21373-tls-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:37:47.193480 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.193482    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvhmq\" (UniqueName: \"kubernetes.io/projected/eed4479f-6670-42f7-b4ce-04b7c7a21373-kube-api-access-pvhmq\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:37:47.193719 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.193491    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed4479f-6670-42f7-b4ce-04b7c7a21373-home\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:37:47.418778 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.418748    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"]
Jul 24 13:37:47.425074 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.425048    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/rate-test-model-b1a3df-kserve-64d5d86bf9-vdp4l"]
Jul 24 13:37:47.950394 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.950279    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7711-139760054543824"]
Jul 24 13:37:47.967142 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.950817    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed4479f-6670-42f7-b4ce-04b7c7a21373" containerName="main"
Jul 24 13:37:47.967142 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.950832    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed4479f-6670-42f7-b4ce-04b7c7a21373" containerName="main"
Jul 24 13:37:47.967142 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.950897    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed4479f-6670-42f7-b4ce-04b7c7a21373" containerName="main"
Jul 24 13:37:47.967142 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.955900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:47.967142 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.959444    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fkqlt\""
Jul 24 13:37:47.967142 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:47.963600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760054543824"]
Jul 24 13:37:48.100858 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:48.100828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54wkb\" (UniqueName: \"kubernetes.io/projected/ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb-kube-api-access-54wkb\") pod \"test-curl-7711-139760054543824\" (UID: \"ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb\") " pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:48.201990 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:48.201892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wkb\" (UniqueName: \"kubernetes.io/projected/ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb-kube-api-access-54wkb\") pod \"test-curl-7711-139760054543824\" (UID: \"ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb\") " pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:48.223804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:48.223775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54wkb\" (UniqueName: \"kubernetes.io/projected/ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb-kube-api-access-54wkb\") pod \"test-curl-7711-139760054543824\" (UID: \"ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb\") " pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:48.265256 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:48.265229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:48.412475 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:48.412451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760054543824"]
Jul 24 13:37:48.414361 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:37:48.414333    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba3de8d8_fba6_4cf6_9055_76d77b5ca7fb.slice/crio-d6a54eaff78d14ad3f677f0a6ce4d9b5f5eba242c31c178a80a9336c35a1b071 WatchSource:0}: Error finding container d6a54eaff78d14ad3f677f0a6ce4d9b5f5eba242c31c178a80a9336c35a1b071: Status 404 returned error can't find the container with id d6a54eaff78d14ad3f677f0a6ce4d9b5f5eba242c31c178a80a9336c35a1b071
Jul 24 13:37:48.415634 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:48.415617    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:37:48.564001 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:48.563969    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed4479f-6670-42f7-b4ce-04b7c7a21373" path="/var/lib/kubelet/pods/eed4479f-6670-42f7-b4ce-04b7c7a21373/volumes"
Jul 24 13:37:49.111117 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:49.111065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760054543824" event={"ID":"ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb","Type":"ContainerStarted","Data":"d6a54eaff78d14ad3f677f0a6ce4d9b5f5eba242c31c178a80a9336c35a1b071"}
Jul 24 13:37:50.118000 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:50.117948    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb" containerID="058e3007987c2cae2a086451ac0861334ae4bafc883c5e640ada4fe23bf6d957" exitCode=0
Jul 24 13:37:50.118406 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:50.118021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760054543824" event={"ID":"ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb","Type":"ContainerDied","Data":"058e3007987c2cae2a086451ac0861334ae4bafc883c5e640ada4fe23bf6d957"}
Jul 24 13:37:51.260993 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:51.260959    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:51.306581 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:51.306557    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7711-139760054543824_ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb/test-curl-7711-139760054543824/0.log"
Jul 24 13:37:51.329213 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:51.329188    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54wkb\" (UniqueName: \"kubernetes.io/projected/ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb-kube-api-access-54wkb\") pod \"ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb\" (UID: \"ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb\") "
Jul 24 13:37:51.331420 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:51.331397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb-kube-api-access-54wkb" (OuterVolumeSpecName: "kube-api-access-54wkb") pod "ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb" (UID: "ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb"). InnerVolumeSpecName "kube-api-access-54wkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:37:51.338347 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:51.338326    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7711-139760054543824"]
Jul 24 13:37:51.345437 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:51.345417    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7711-139760054543824"]
Jul 24 13:37:51.430161 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:51.430141    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54wkb\" (UniqueName: \"kubernetes.io/projected/ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb-kube-api-access-54wkb\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:37:52.130221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.130195    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:52.130221 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.130224    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6a54eaff78d14ad3f677f0a6ce4d9b5f5eba242c31c178a80a9336c35a1b071"
Jul 24 13:37:52.139867 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:37:52.139841    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-139760054543824\" is forbidden: User \"system:node:ip-10-0-141-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-29.ec2.internal' and this object" podUID="ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb" pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:52.190670 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.190644    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"]
Jul 24 13:37:52.191051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.191037    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb" containerName="test-curl-7711-139760054543824"
Jul 24 13:37:52.191051 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.191051    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb" containerName="test-curl-7711-139760054543824"
Jul 24 13:37:52.191201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.191125    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb" containerName="test-curl-7711-139760054543824"
Jul 24 13:37:52.194249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.194232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.196676 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:37:52.196648    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-139760054543824\" is forbidden: User \"system:node:ip-10-0-141-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-29.ec2.internal' and this object" podUID="ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb" pod="opendatahub/test-curl-7711-139760054543824"
Jul 24 13:37:52.197032 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.197012    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-49d475-openshift-default-dockercfg-mtnhf\""
Jul 24 13:37:52.212724 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.212704    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"]
Jul 24 13:37:52.338046 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-credential-socket\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.338396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68797318-58fd-41d1-9b61-77930b4dcd47-istiod-ca-cert\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.338396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-certs\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.338396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-envoy\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.338396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68797318-58fd-41d1-9b61-77930b4dcd47-istio-podinfo\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.338396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-istio-token\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.338396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-socket\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.338396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnck\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-kube-api-access-zcnck\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.338396 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.338377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-data\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.439748 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.439724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-credential-socket\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.439939 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.439762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68797318-58fd-41d1-9b61-77930b4dcd47-istiod-ca-cert\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.439939 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.439779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-certs\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.439939 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.439910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-envoy\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440119 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.439976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68797318-58fd-41d1-9b61-77930b4dcd47-istio-podinfo\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440119 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-istio-token\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440119 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-socket\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440119 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnck\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-kube-api-access-zcnck\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440119 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-data\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440306 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-certs\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440430 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-credential-socket\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440559 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-socket\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440632 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-data\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.440736 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.440717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68797318-58fd-41d1-9b61-77930b4dcd47-istiod-ca-cert\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.442348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.442325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-envoy\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.442544 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.442527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68797318-58fd-41d1-9b61-77930b4dcd47-istio-podinfo\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.450979 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.450958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-istio-token\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.451154 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.451137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnck\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-kube-api-access-zcnck\") pod \"e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") " pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.504467 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.504426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:52.565645 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.565616    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb" path="/var/lib/kubelet/pods/ba3de8d8-fba6-4cf6-9055-76d77b5ca7fb/volumes"
Jul 24 13:37:52.655036 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.655008    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"]
Jul 24 13:37:52.656617 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:37:52.656594    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68797318_58fd_41d1_9b61_77930b4dcd47.slice/crio-04bd2b6e7ccc8b753c11b9da3199e2b9db840dee60fa8602850d6a719c48f8b5 WatchSource:0}: Error finding container 04bd2b6e7ccc8b753c11b9da3199e2b9db840dee60fa8602850d6a719c48f8b5: Status 404 returned error can't find the container with id 04bd2b6e7ccc8b753c11b9da3199e2b9db840dee60fa8602850d6a719c48f8b5
Jul 24 13:37:52.659031 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.659000    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:37:52.659106 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.659064    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:37:52.659106 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:52.659091    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:37:53.137159 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:53.137075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" event={"ID":"68797318-58fd-41d1-9b61-77930b4dcd47","Type":"ContainerStarted","Data":"2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3"}
Jul 24 13:37:53.137159 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:53.137134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" event={"ID":"68797318-58fd-41d1-9b61-77930b4dcd47","Type":"ContainerStarted","Data":"04bd2b6e7ccc8b753c11b9da3199e2b9db840dee60fa8602850d6a719c48f8b5"}
Jul 24 13:37:53.170353 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:53.170305    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" podStartSLOduration=1.170291207 podStartE2EDuration="1.170291207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:37:53.168886306 +0000 UTC m=+2697.158205003" watchObservedRunningTime="2026-07-24 13:37:53.170291207 +0000 UTC m=+2697.159609902"
Jul 24 13:37:53.505011 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:53.504984    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:53.509848 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:53.509822    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:54.146806 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:54.146060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:54.149566 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:54.149538    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:37:55.387565 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.387525    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"]
Jul 24 13:37:55.391938 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.391894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:55.397070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.397049    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-49d475-serving-cert\""
Jul 24 13:37:55.404391 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.404370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"]
Jul 24 13:37:55.465685 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.465657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b369ab-ac94-4642-862d-23c8bbb1b1c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:55.465831 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.465696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:55.465831 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.465778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddjr\" (UniqueName: \"kubernetes.io/projected/08b369ab-ac94-4642-862d-23c8bbb1b1c2-kube-api-access-5ddjr\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:55.566526 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.566486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddjr\" (UniqueName: \"kubernetes.io/projected/08b369ab-ac94-4642-862d-23c8bbb1b1c2-kube-api-access-5ddjr\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:55.566719 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.566550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b369ab-ac94-4642-862d-23c8bbb1b1c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:55.566719 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.566577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:55.566719 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:37:55.566672    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-serving-cert: secret "maas-api-e2e-ipp-a-49d475-serving-cert" not found
Jul 24 13:37:55.566894 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:37:55.566726    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls podName:08b369ab-ac94-4642-862d-23c8bbb1b1c2 nodeName:}" failed. No retries permitted until 2026-07-24 13:37:56.066711668 +0000 UTC m=+2700.056030342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls") pod "maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p" (UID: "08b369ab-ac94-4642-862d-23c8bbb1b1c2") : secret "maas-api-e2e-ipp-a-49d475-serving-cert" not found
Jul 24 13:37:55.567277 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.567255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b369ab-ac94-4642-862d-23c8bbb1b1c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:55.577303 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:55.577273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddjr\" (UniqueName: \"kubernetes.io/projected/08b369ab-ac94-4642-862d-23c8bbb1b1c2-kube-api-access-5ddjr\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:56.071562 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:56.071535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:56.074114 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:56.074089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:56.303532 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:56.303504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:56.671249 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:56.671215    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"]
Jul 24 13:37:56.673133 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:37:56.673096    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b369ab_ac94_4642_862d_23c8bbb1b1c2.slice/crio-cf9b1011a3c8b0c8fe963d4849574940d1c7cae6ff75266b6ce5ddc1ee6f6573 WatchSource:0}: Error finding container cf9b1011a3c8b0c8fe963d4849574940d1c7cae6ff75266b6ce5ddc1ee6f6573: Status 404 returned error can't find the container with id cf9b1011a3c8b0c8fe963d4849574940d1c7cae6ff75266b6ce5ddc1ee6f6573
Jul 24 13:37:57.162405 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:57.162376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p" event={"ID":"08b369ab-ac94-4642-862d-23c8bbb1b1c2","Type":"ContainerStarted","Data":"cf9b1011a3c8b0c8fe963d4849574940d1c7cae6ff75266b6ce5ddc1ee6f6573"}
Jul 24 13:37:58.169114 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:58.169071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p" event={"ID":"08b369ab-ac94-4642-862d-23c8bbb1b1c2","Type":"ContainerStarted","Data":"e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84"}
Jul 24 13:37:58.169524 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:58.169272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:37:58.198777 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:37:58.198726    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p" podStartSLOduration=2.740485311 podStartE2EDuration="3.198711892s" totalImagesPullingTime="458.226581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:37:58.197602854 +0000 UTC m=+2702.186921561" watchObservedRunningTime="2026-07-24 13:37:58.198711892 +0000 UTC m=+2702.188030587"
Jul 24 13:38:04.182891 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:04.182858    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:38:10.188156 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:10.188101    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 24 13:38:10.188630 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:10.188194    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 13:38:11.252802 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:11.252775    2579 generic.go:356] "Generic (PLEG): container finished" podID="68797318-58fd-41d1-9b61-77930b4dcd47" containerID="2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3" exitCode=0
Jul 24 13:38:11.253162 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:11.252810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" event={"ID":"68797318-58fd-41d1-9b61-77930b4dcd47","Type":"ContainerDied","Data":"2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3"}
Jul 24 13:38:11.253206 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:11.253175    2579 scope.go:117] "RemoveContainer" containerID="2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3"
Jul 24 13:38:11.254185 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:11.254159    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:38:11.254247 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:11.254235    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:38:11.254302 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:11.254289    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:38:12.261085 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:12.261047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" event={"ID":"68797318-58fd-41d1-9b61-77930b4dcd47","Type":"ContainerStarted","Data":"8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe"}
Jul 24 13:38:12.505216 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:12.505177    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:38:12.505422 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:12.505307    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Jul 24 13:38:12.505422 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:12.505364    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Jul 24 13:38:13.505769 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:13.505726    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Jul 24 13:38:13.506253 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:13.505791    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Jul 24 13:38:14.505503 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:14.505459    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Jul 24 13:38:14.505825 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:14.505775    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Jul 24 13:38:15.505908 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:15.505869    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Jul 24 13:38:15.506506 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:15.506475    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Jul 24 13:38:16.508736 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:16.508697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:38:16.509229 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:16.509098    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:38:16.509655 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:16.509634    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:38:45.611917 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.611877    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"]
Jul 24 13:38:45.615670 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.615648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.619177 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.619154    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-49d475\"/\"kube-root-ca.crt\""
Jul 24 13:38:45.619314 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.619154    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-49d475\"/\"default-dockercfg-kg8jn\""
Jul 24 13:38:45.619314 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.619187    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-49d475\"/\"openshift-service-ca.crt\""
Jul 24 13:38:45.619314 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.619154    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-49d475\"/\"ipp-route-49d475-kserve-self-signed-certs\""
Jul 24 13:38:45.625860 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.625837    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"]
Jul 24 13:38:45.751376 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.751339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-dshm\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.751376 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.751380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-home\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.751595 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.751396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tmp-dir\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.751595 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.751421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdg2g\" (UniqueName: \"kubernetes.io/projected/671d15b6-39d4-4c5a-b4c7-86ceff680bda-kube-api-access-tdg2g\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.751595 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.751482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-model-cache\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.751595 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.751504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tls-certs\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.852506 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.852471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tls-certs\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.852687 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.852530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-dshm\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.852687 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.852561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-home\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.852687 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.852579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tmp-dir\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.852687 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.852601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdg2g\" (UniqueName: \"kubernetes.io/projected/671d15b6-39d4-4c5a-b4c7-86ceff680bda-kube-api-access-tdg2g\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.852687 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.852683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-model-cache\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.853076 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.853055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-home\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.853162 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.853072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-model-cache\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.853162 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.853117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tmp-dir\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.855057 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.855033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-dshm\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.855220 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.855203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tls-certs\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.861034 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.861006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdg2g\" (UniqueName: \"kubernetes.io/projected/671d15b6-39d4-4c5a-b4c7-86ceff680bda-kube-api-access-tdg2g\") pod \"ipp-route-49d475-kserve-845557f6cf-rw4v8\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") " pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:45.925332 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:45.925290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:46.067330 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:46.067306    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"]
Jul 24 13:38:46.068717 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:38:46.068683    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod671d15b6_39d4_4c5a_b4c7_86ceff680bda.slice/crio-a4dacd736e78e01b9db4818b32ad820fc9a32fe1c7e07086122bb50bda782d49 WatchSource:0}: Error finding container a4dacd736e78e01b9db4818b32ad820fc9a32fe1c7e07086122bb50bda782d49: Status 404 returned error can't find the container with id a4dacd736e78e01b9db4818b32ad820fc9a32fe1c7e07086122bb50bda782d49
Jul 24 13:38:46.460117 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:46.460085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8" event={"ID":"671d15b6-39d4-4c5a-b4c7-86ceff680bda","Type":"ContainerStarted","Data":"18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7"}
Jul 24 13:38:46.460289 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:46.460127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8" event={"ID":"671d15b6-39d4-4c5a-b4c7-86ceff680bda","Type":"ContainerStarted","Data":"a4dacd736e78e01b9db4818b32ad820fc9a32fe1c7e07086122bb50bda782d49"}
Jul 24 13:38:46.481288 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:46.481243    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8" podStartSLOduration=1.481229074 podStartE2EDuration="1.481229074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:46.477672264 +0000 UTC m=+2750.466990972" watchObservedRunningTime="2026-07-24 13:38:46.481229074 +0000 UTC m=+2750.470547770"
Jul 24 13:38:55.926340 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:55.926301    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:55.926340 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:55.926345    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:55.938565 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:55.938542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:38:56.529754 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:38:56.529720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:39:16.136040 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.136006    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"]
Jul 24 13:39:16.139551 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.139533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.142197 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.142178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-45ac1f-openshift-default-dockercfg-zz6h2\""
Jul 24 13:39:16.154350 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.154322    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"]
Jul 24 13:39:16.258785 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.258741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-certs\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.259118 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.259092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d93e51-5a27-4306-b6c5-c06f0f682027-istiod-ca-cert\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.259355 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.259337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-envoy\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.259476 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.259461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-podinfo\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.259608 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.259583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-data\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.259734 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.259630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-credential-socket\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.259734 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.259655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-token\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.259822 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.259732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-socket\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.259870 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.259811    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv947\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-kube-api-access-wv947\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.360659 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-data\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.360659 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-credential-socket\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.360893 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-token\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.360893 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-socket\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.360893 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv947\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-kube-api-access-wv947\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.360893 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-certs\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.360893 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d93e51-5a27-4306-b6c5-c06f0f682027-istiod-ca-cert\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.360893 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-envoy\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.361226 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.360936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-podinfo\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.361226 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.361081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-data\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.361226 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.361093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-credential-socket\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.361226 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.361196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-socket\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.361427 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.361236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-certs\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.361514 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.361497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d93e51-5a27-4306-b6c5-c06f0f682027-istiod-ca-cert\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.363324 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.363303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-envoy\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.363520 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.363504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-podinfo\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.369337 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.369313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-token\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.369485 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.369470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv947\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-kube-api-access-wv947\") pod \"e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") " pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.449535 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.449505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:16.602505 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.602470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"]
Jul 24 13:39:16.603262 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:39:16.603234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d93e51_5a27_4306_b6c5_c06f0f682027.slice/crio-124d0f501654bd2c443a7db9fdceebf38c35f5dd92eff40744ab63829f704114 WatchSource:0}: Error finding container 124d0f501654bd2c443a7db9fdceebf38c35f5dd92eff40744ab63829f704114: Status 404 returned error can't find the container with id 124d0f501654bd2c443a7db9fdceebf38c35f5dd92eff40744ab63829f704114
Jul 24 13:39:16.606154 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.606116    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:39:16.606296 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.606202    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:39:16.606296 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.606241    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:39:16.632871 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:16.632845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" event={"ID":"c0d93e51-5a27-4306-b6c5-c06f0f682027","Type":"ContainerStarted","Data":"124d0f501654bd2c443a7db9fdceebf38c35f5dd92eff40744ab63829f704114"}
Jul 24 13:39:17.640070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:17.640033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" event={"ID":"c0d93e51-5a27-4306-b6c5-c06f0f682027","Type":"ContainerStarted","Data":"25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51"}
Jul 24 13:39:17.665768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:17.665710    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" podStartSLOduration=1.6656956520000001 podStartE2EDuration="1.665695652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:17.662736714 +0000 UTC m=+2781.652055408" watchObservedRunningTime="2026-07-24 13:39:17.665695652 +0000 UTC m=+2781.655014347"
Jul 24 13:39:18.450022 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:18.449990    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:18.455018 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:18.454996    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:18.646383 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:18.646351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:18.647383 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:18.647364    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:38.770828 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:38.770785    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerID="25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51" exitCode=0
Jul 24 13:39:38.771377 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:38.770841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" event={"ID":"c0d93e51-5a27-4306-b6c5-c06f0f682027","Type":"ContainerDied","Data":"25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51"}
Jul 24 13:39:38.771439 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:38.771421    2579 scope.go:117] "RemoveContainer" containerID="25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51"
Jul 24 13:39:38.772680 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:38.772657    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:39:38.772735 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:38.772723    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:39:38.772770 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:38.772755    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 24 13:39:39.777934 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:39.777888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" event={"ID":"c0d93e51-5a27-4306-b6c5-c06f0f682027","Type":"ContainerStarted","Data":"9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e"}
Jul 24 13:39:40.450180 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:40.450139    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:40.450358 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:40.450329    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Jul 24 13:39:40.450408 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:40.450375    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Jul 24 13:39:41.449852 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:41.449819    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Jul 24 13:39:41.450256 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:41.449869    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Jul 24 13:39:42.450289 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:42.450257    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Jul 24 13:39:42.450693 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:42.450314    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Jul 24 13:39:43.449848 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:43.449817    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Jul 24 13:39:43.450041 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:43.449877    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Jul 24 13:39:44.453937 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:44.453891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:44.454373 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:44.454354    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:44.455107 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:44.455084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:39:57.576253 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:57.576216    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"]
Jul 24 13:39:57.576773 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:57.576464    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy" containerID="cri-o://9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e" gracePeriod=30
Jul 24 13:39:59.457407 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:59.457371    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 13:39:59.457785 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:39:59.457428    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 13:40:02.832585 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.832557    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:40:02.915498 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.915469    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerID="9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e" exitCode=0
Jul 24 13:40:02.915663 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.915518    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"
Jul 24 13:40:02.915663 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.915547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" event={"ID":"c0d93e51-5a27-4306-b6c5-c06f0f682027","Type":"ContainerDied","Data":"9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e"}
Jul 24 13:40:02.915663 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.915605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb" event={"ID":"c0d93e51-5a27-4306-b6c5-c06f0f682027","Type":"ContainerDied","Data":"124d0f501654bd2c443a7db9fdceebf38c35f5dd92eff40744ab63829f704114"}
Jul 24 13:40:02.915663 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.915635    2579 scope.go:117] "RemoveContainer" containerID="9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e"
Jul 24 13:40:02.924143 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.924115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-podinfo\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.924303 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.924153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-token\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.924303 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.924179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-envoy\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.924303 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.924243    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv947\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-kube-api-access-wv947\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.924473 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.924362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-credential-socket\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.924695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.924654    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:02.924902 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.924885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-certs\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.925321 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.925306    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-socket\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.925512 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.925246    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:02.925587 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.925492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d93e51-5a27-4306-b6c5-c06f0f682027-istiod-ca-cert\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.925644 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.925601    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-data\") pod \"c0d93e51-5a27-4306-b6c5-c06f0f682027\" (UID: \"c0d93e51-5a27-4306-b6c5-c06f0f682027\") "
Jul 24 13:40:02.926071 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.925996    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:02.926071 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.926017    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:02.926197 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.926163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:02.926255 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.926208    2579 scope.go:117] "RemoveContainer" containerID="25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51"
Jul 24 13:40:02.926353 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.926330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d93e51-5a27-4306-b6c5-c06f0f682027-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:40:02.926884 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.926860    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-kube-api-access-wv947" (OuterVolumeSpecName: "kube-api-access-wv947") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "kube-api-access-wv947". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:40:02.927599 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.927559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:40:02.927599 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.927565    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:40:02.927599 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.927570    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:02.927599 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.927580    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0d93e51-5a27-4306-b6c5-c06f0f682027" (UID: "c0d93e51-5a27-4306-b6c5-c06f0f682027"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:02.948762 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.948715    2579 scope.go:117] "RemoveContainer" containerID="9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e"
Jul 24 13:40:02.949065 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:40:02.949045    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e\": container with ID starting with 9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e not found: ID does not exist" containerID="9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e"
Jul 24 13:40:02.949165 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.949074    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e"} err="failed to get container status \"9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e\": rpc error: code = NotFound desc = could not find container \"9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e\": container with ID starting with 9fb0ea0ae6fb56a0f74c8adc238f6629b484b4596d9ed1debc6ac940d0d7af4e not found: ID does not exist"
Jul 24 13:40:02.949165 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.949095    2579 scope.go:117] "RemoveContainer" containerID="25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51"
Jul 24 13:40:02.949308 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:40:02.949289    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51\": container with ID starting with 25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51 not found: ID does not exist" containerID="25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51"
Jul 24 13:40:02.949363 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:02.949318    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51"} err="failed to get container status \"25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51\": rpc error: code = NotFound desc = could not find container \"25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51\": container with ID starting with 25867472f392cffae6c1a200ea5a5f442e84c2b3b7b39c85532fe9a9746eee51 not found: ID does not exist"
Jul 24 13:40:03.027103 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.027072    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:03.027103 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.027096    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d93e51-5a27-4306-b6c5-c06f0f682027-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:03.027103 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.027105    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:03.027103 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.027113    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:03.027362 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.027121    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:03.027362 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.027129    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d93e51-5a27-4306-b6c5-c06f0f682027-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:03.027362 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.027137    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv947\" (UniqueName: \"kubernetes.io/projected/c0d93e51-5a27-4306-b6c5-c06f0f682027-kube-api-access-wv947\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:03.239488 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.239449    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"]
Jul 24 13:40:03.243829 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:03.243804    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-45ac1f-openshift-default-6bcf456ff6-psszb"]
Jul 24 13:40:04.567380 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:04.567342    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" path="/var/lib/kubelet/pods/c0d93e51-5a27-4306-b6c5-c06f0f682027/volumes"
Jul 24 13:40:18.352682 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.352649    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"]
Jul 24 13:40:18.353180 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.352976    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p" podUID="08b369ab-ac94-4642-862d-23c8bbb1b1c2" containerName="maas-api" containerID="cri-o://e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84" gracePeriod=30
Jul 24 13:40:18.626247 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.626165    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:40:18.669580 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.669554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ddjr\" (UniqueName: \"kubernetes.io/projected/08b369ab-ac94-4642-862d-23c8bbb1b1c2-kube-api-access-5ddjr\") pod \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") "
Jul 24 13:40:18.669768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.669607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls\") pod \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") "
Jul 24 13:40:18.669768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.669716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b369ab-ac94-4642-862d-23c8bbb1b1c2-openshift-service-ca\") pod \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\" (UID: \"08b369ab-ac94-4642-862d-23c8bbb1b1c2\") "
Jul 24 13:40:18.670328 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.670293    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b369ab-ac94-4642-862d-23c8bbb1b1c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08b369ab-ac94-4642-862d-23c8bbb1b1c2" (UID: "08b369ab-ac94-4642-862d-23c8bbb1b1c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:40:18.672504 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.672477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08b369ab-ac94-4642-862d-23c8bbb1b1c2" (UID: "08b369ab-ac94-4642-862d-23c8bbb1b1c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:40:18.672504 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.672496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b369ab-ac94-4642-862d-23c8bbb1b1c2-kube-api-access-5ddjr" (OuterVolumeSpecName: "kube-api-access-5ddjr") pod "08b369ab-ac94-4642-862d-23c8bbb1b1c2" (UID: "08b369ab-ac94-4642-862d-23c8bbb1b1c2"). InnerVolumeSpecName "kube-api-access-5ddjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:40:18.770638 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.770602    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b369ab-ac94-4642-862d-23c8bbb1b1c2-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:18.770638 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.770630    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ddjr\" (UniqueName: \"kubernetes.io/projected/08b369ab-ac94-4642-862d-23c8bbb1b1c2-kube-api-access-5ddjr\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:18.770638 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:18.770640    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b369ab-ac94-4642-862d-23c8bbb1b1c2-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:19.008347 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.008313    2579 generic.go:356] "Generic (PLEG): container finished" podID="08b369ab-ac94-4642-862d-23c8bbb1b1c2" containerID="e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84" exitCode=0
Jul 24 13:40:19.008513 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.008354    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"
Jul 24 13:40:19.008513 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.008391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p" event={"ID":"08b369ab-ac94-4642-862d-23c8bbb1b1c2","Type":"ContainerDied","Data":"e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84"}
Jul 24 13:40:19.008513 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.008435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p" event={"ID":"08b369ab-ac94-4642-862d-23c8bbb1b1c2","Type":"ContainerDied","Data":"cf9b1011a3c8b0c8fe963d4849574940d1c7cae6ff75266b6ce5ddc1ee6f6573"}
Jul 24 13:40:19.008513 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.008455    2579 scope.go:117] "RemoveContainer" containerID="e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84"
Jul 24 13:40:19.018319 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.018301    2579 scope.go:117] "RemoveContainer" containerID="e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84"
Jul 24 13:40:19.018573 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:40:19.018551    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84\": container with ID starting with e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84 not found: ID does not exist" containerID="e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84"
Jul 24 13:40:19.018643 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.018584    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84"} err="failed to get container status \"e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84\": rpc error: code = NotFound desc = could not find container \"e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84\": container with ID starting with e193e48ed6eb3844f21b5a3e2e024a1e41d830d7e7f9175913a490f989fbea84 not found: ID does not exist"
Jul 24 13:40:19.037135 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.035538    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"]
Jul 24 13:40:19.040524 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:19.040502    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49d475-758fffc9dc-c296p"]
Jul 24 13:40:20.563714 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:20.563673    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b369ab-ac94-4642-862d-23c8bbb1b1c2" path="/var/lib/kubelet/pods/08b369ab-ac94-4642-862d-23c8bbb1b1c2/volumes"
Jul 24 13:40:26.850500 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:26.850461    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"]
Jul 24 13:40:41.870084 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:41.870020    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8" podUID="671d15b6-39d4-4c5a-b4c7-86ceff680bda" containerName="main" containerID="cri-o://18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7" gracePeriod=45
Jul 24 13:40:42.129426 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.129370    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:40:42.138842 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.138811    2579 generic.go:356] "Generic (PLEG): container finished" podID="671d15b6-39d4-4c5a-b4c7-86ceff680bda" containerID="18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7" exitCode=0
Jul 24 13:40:42.138989 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.138847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"
Jul 24 13:40:42.138989 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.138889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8" event={"ID":"671d15b6-39d4-4c5a-b4c7-86ceff680bda","Type":"ContainerDied","Data":"18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7"}
Jul 24 13:40:42.138989 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.138953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8" event={"ID":"671d15b6-39d4-4c5a-b4c7-86ceff680bda","Type":"ContainerDied","Data":"a4dacd736e78e01b9db4818b32ad820fc9a32fe1c7e07086122bb50bda782d49"}
Jul 24 13:40:42.138989 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.138975    2579 scope.go:117] "RemoveContainer" containerID="18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7"
Jul 24 13:40:42.147340 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.147320    2579 scope.go:117] "RemoveContainer" containerID="18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7"
Jul 24 13:40:42.147578 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:40:42.147560    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7\": container with ID starting with 18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7 not found: ID does not exist" containerID="18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7"
Jul 24 13:40:42.147648 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.147588    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7"} err="failed to get container status \"18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7\": rpc error: code = NotFound desc = could not find container \"18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7\": container with ID starting with 18373fd0f51640454a322155717eb08383c21ae11a210e3b4c947ea465cc1ea7 not found: ID does not exist"
Jul 24 13:40:42.308541 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.308510    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-home\") pod \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") "
Jul 24 13:40:42.308541 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.308543    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdg2g\" (UniqueName: \"kubernetes.io/projected/671d15b6-39d4-4c5a-b4c7-86ceff680bda-kube-api-access-tdg2g\") pod \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") "
Jul 24 13:40:42.308804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.308605    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tls-certs\") pod \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") "
Jul 24 13:40:42.308804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.308638    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-dshm\") pod \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") "
Jul 24 13:40:42.308804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.308653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-model-cache\") pod \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") "
Jul 24 13:40:42.308804 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.308714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tmp-dir\") pod \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\" (UID: \"671d15b6-39d4-4c5a-b4c7-86ceff680bda\") "
Jul 24 13:40:42.309050 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.308870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-home" (OuterVolumeSpecName: "home") pod "671d15b6-39d4-4c5a-b4c7-86ceff680bda" (UID: "671d15b6-39d4-4c5a-b4c7-86ceff680bda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:42.309103 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.309044    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-model-cache" (OuterVolumeSpecName: "model-cache") pod "671d15b6-39d4-4c5a-b4c7-86ceff680bda" (UID: "671d15b6-39d4-4c5a-b4c7-86ceff680bda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:42.309103 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.309059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "671d15b6-39d4-4c5a-b4c7-86ceff680bda" (UID: "671d15b6-39d4-4c5a-b4c7-86ceff680bda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:42.309207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.309145    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-home\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:42.311067 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.311045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "671d15b6-39d4-4c5a-b4c7-86ceff680bda" (UID: "671d15b6-39d4-4c5a-b4c7-86ceff680bda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:40:42.311164 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.311080    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-dshm" (OuterVolumeSpecName: "dshm") pod "671d15b6-39d4-4c5a-b4c7-86ceff680bda" (UID: "671d15b6-39d4-4c5a-b4c7-86ceff680bda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:40:42.311164 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.311114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671d15b6-39d4-4c5a-b4c7-86ceff680bda-kube-api-access-tdg2g" (OuterVolumeSpecName: "kube-api-access-tdg2g") pod "671d15b6-39d4-4c5a-b4c7-86ceff680bda" (UID: "671d15b6-39d4-4c5a-b4c7-86ceff680bda"). InnerVolumeSpecName "kube-api-access-tdg2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:40:42.409839 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.409812    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-dshm\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:42.409839 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.409837    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-model-cache\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:42.410050 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.409851    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tmp-dir\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:42.410050 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.409863    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdg2g\" (UniqueName: \"kubernetes.io/projected/671d15b6-39d4-4c5a-b4c7-86ceff680bda-kube-api-access-tdg2g\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:42.410050 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.409874    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671d15b6-39d4-4c5a-b4c7-86ceff680bda-tls-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:40:42.465682 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.465652    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"]
Jul 24 13:40:42.471891 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.471865    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-49d475/ipp-route-49d475-kserve-845557f6cf-rw4v8"]
Jul 24 13:40:42.564574 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:40:42.564535    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="671d15b6-39d4-4c5a-b4c7-86ceff680bda" path="/var/lib/kubelet/pods/671d15b6-39d4-4c5a-b4c7-86ceff680bda/volumes"
Jul 24 13:41:11.914989 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:11.914950    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"]
Jul 24 13:41:11.915517 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:11.915203    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy" containerID="cri-o://8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe" gracePeriod=30
Jul 24 13:41:16.511213 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:16.511177    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 13:41:16.511576 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:16.511225    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 13:41:17.178743 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.178689    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:41:17.336877 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.336844    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-certs\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.336877 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.336876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcnck\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-kube-api-access-zcnck\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.337121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.336913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-envoy\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.337121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.336960    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-data\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.337121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68797318-58fd-41d1-9b61-77930b4dcd47-istiod-ca-cert\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.337121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337043    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68797318-58fd-41d1-9b61-77930b4dcd47-istio-podinfo\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.337121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-socket\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.337121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337088    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-credential-socket\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.337121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-istio-token\") pod \"68797318-58fd-41d1-9b61-77930b4dcd47\" (UID: \"68797318-58fd-41d1-9b61-77930b4dcd47\") "
Jul 24 13:41:17.337416 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:41:17.337456 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:41:17.337518 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68797318-58fd-41d1-9b61-77930b4dcd47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:41:17.337575 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337528    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:17.337626 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337573    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:17.337626 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:41:17.338034 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.337975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-data" (OuterVolumeSpecName: "istio-data") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:41:17.339430 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.339399    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:41:17.339430 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.339396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68797318-58fd-41d1-9b61-77930b4dcd47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:41:17.339579 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.339438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-istio-token" (OuterVolumeSpecName: "istio-token") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:41:17.339769 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.339740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-kube-api-access-zcnck" (OuterVolumeSpecName: "kube-api-access-zcnck") pod "68797318-58fd-41d1-9b61-77930b4dcd47" (UID: "68797318-58fd-41d1-9b61-77930b4dcd47"). InnerVolumeSpecName "kube-api-access-zcnck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:41:17.346602 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.346526    2579 generic.go:356] "Generic (PLEG): container finished" podID="68797318-58fd-41d1-9b61-77930b4dcd47" containerID="8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe" exitCode=0
Jul 24 13:41:17.346649 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.346628    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"
Jul 24 13:41:17.346769 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.346657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" event={"ID":"68797318-58fd-41d1-9b61-77930b4dcd47","Type":"ContainerDied","Data":"8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe"}
Jul 24 13:41:17.346769 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.346731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6" event={"ID":"68797318-58fd-41d1-9b61-77930b4dcd47","Type":"ContainerDied","Data":"04bd2b6e7ccc8b753c11b9da3199e2b9db840dee60fa8602850d6a719c48f8b5"}
Jul 24 13:41:17.346769 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.346759    2579 scope.go:117] "RemoveContainer" containerID="8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe"
Jul 24 13:41:17.361216 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.361200    2579 scope.go:117] "RemoveContainer" containerID="2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3"
Jul 24 13:41:17.376133 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.376115    2579 scope.go:117] "RemoveContainer" containerID="8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe"
Jul 24 13:41:17.376387 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:41:17.376368    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe\": container with ID starting with 8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe not found: ID does not exist" containerID="8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe"
Jul 24 13:41:17.376447 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.376394    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe"} err="failed to get container status \"8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe\": rpc error: code = NotFound desc = could not find container \"8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe\": container with ID starting with 8b1e5319904a355ce81572930d1d17f1096b0b85ba18d114978849202b49e2fe not found: ID does not exist"
Jul 24 13:41:17.376447 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.376412    2579 scope.go:117] "RemoveContainer" containerID="2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3"
Jul 24 13:41:17.376650 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:41:17.376635    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3\": container with ID starting with 2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3 not found: ID does not exist" containerID="2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3"
Jul 24 13:41:17.376690 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.376656    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3"} err="failed to get container status \"2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3\": rpc error: code = NotFound desc = could not find container \"2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3\": container with ID starting with 2a97e9f29e000e2a19eb89b7ae817384c31bf9289a4ee9960310f1ca239f49f3 not found: ID does not exist"
Jul 24 13:41:17.380415 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.380393    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"]
Jul 24 13:41:17.384769 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.384750    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-49d475-openshift-default-5f478d5464-mq6m6"]
Jul 24 13:41:17.438556 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.438498    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcnck\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-kube-api-access-zcnck\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:17.438556 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.438523    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:17.438556 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.438536    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:17.438556 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.438547    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68797318-58fd-41d1-9b61-77930b4dcd47-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:17.438556 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.438558    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68797318-58fd-41d1-9b61-77930b4dcd47-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:17.438766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.438569    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68797318-58fd-41d1-9b61-77930b4dcd47-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:17.438766 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:17.438579    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68797318-58fd-41d1-9b61-77930b4dcd47-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:18.564280 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:18.564249    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" path="/var/lib/kubelet/pods/68797318-58fd-41d1-9b61-77930b4dcd47/volumes"
Jul 24 13:41:45.859684 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:45.859653    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"]
Jul 24 13:41:45.860178 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:45.859912    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7" podUID="4a8fbe7c-4288-431e-bd01-bdd54bf12d56" containerName="maas-api" containerID="cri-o://b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a" gracePeriod=30
Jul 24 13:41:46.113640 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.113531    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:41:46.205527 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.205486    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls\") pod \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") "
Jul 24 13:41:46.205704 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.205614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxr7n\" (UniqueName: \"kubernetes.io/projected/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-kube-api-access-wxr7n\") pod \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") "
Jul 24 13:41:46.205704 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.205658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-openshift-service-ca\") pod \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\" (UID: \"4a8fbe7c-4288-431e-bd01-bdd54bf12d56\") "
Jul 24 13:41:46.206094 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.206066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a8fbe7c-4288-431e-bd01-bdd54bf12d56" (UID: "4a8fbe7c-4288-431e-bd01-bdd54bf12d56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:41:46.207850 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.207821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a8fbe7c-4288-431e-bd01-bdd54bf12d56" (UID: "4a8fbe7c-4288-431e-bd01-bdd54bf12d56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:41:46.207850 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.207835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-kube-api-access-wxr7n" (OuterVolumeSpecName: "kube-api-access-wxr7n") pod "4a8fbe7c-4288-431e-bd01-bdd54bf12d56" (UID: "4a8fbe7c-4288-431e-bd01-bdd54bf12d56"). InnerVolumeSpecName "kube-api-access-wxr7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:41:46.307354 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.307322    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxr7n\" (UniqueName: \"kubernetes.io/projected/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-kube-api-access-wxr7n\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:46.307354 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.307350    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:46.307354 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.307359    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8fbe7c-4288-431e-bd01-bdd54bf12d56-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:41:46.505874 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.505835    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a8fbe7c-4288-431e-bd01-bdd54bf12d56" containerID="b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a" exitCode=0
Jul 24 13:41:46.506071 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.505883    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"
Jul 24 13:41:46.506071 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.505895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7" event={"ID":"4a8fbe7c-4288-431e-bd01-bdd54bf12d56","Type":"ContainerDied","Data":"b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a"}
Jul 24 13:41:46.506071 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.505968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7" event={"ID":"4a8fbe7c-4288-431e-bd01-bdd54bf12d56","Type":"ContainerDied","Data":"6b308e8700b38e424af5bd5ebf1db722551eefc158aa72e43c0e97ac87fde7aa"}
Jul 24 13:41:46.506071 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.505998    2579 scope.go:117] "RemoveContainer" containerID="b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a"
Jul 24 13:41:46.515491 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.515473    2579 scope.go:117] "RemoveContainer" containerID="b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a"
Jul 24 13:41:46.515735 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:41:46.515714    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a\": container with ID starting with b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a not found: ID does not exist" containerID="b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a"
Jul 24 13:41:46.515785 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.515745    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a"} err="failed to get container status \"b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a\": rpc error: code = NotFound desc = could not find container \"b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a\": container with ID starting with b93ecc6059f54f2786a005482dea14efea55b0e176d90af8e7c877ceb6eb3b3a not found: ID does not exist"
Jul 24 13:41:46.529335 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.529310    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"]
Jul 24 13:41:46.534176 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.534157    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f3a913-6ccbbb6d47-c9ck7"]
Jul 24 13:41:46.563792 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:46.563766    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8fbe7c-4288-431e-bd01-bdd54bf12d56" path="/var/lib/kubelet/pods/4a8fbe7c-4288-431e-bd01-bdd54bf12d56/volumes"
Jul 24 13:41:57.790512 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:57.790476    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"]
Jul 24 13:41:57.791007 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:41:57.790678    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy" containerID="cri-o://453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82" gracePeriod=30
Jul 24 13:42:02.601587 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:02.601537    2579 patch_prober.go:28] interesting pod/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 13:42:02.601982 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:02.601635    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 13:42:03.047728 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.047704    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:42:03.160536 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-envoy\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.160536 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istiod-ca-cert\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.160759 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-credential-socket\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.160759 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160600    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-data\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.160759 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-token\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.160759 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-socket\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.161009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swhvq\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-kube-api-access-swhvq\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.161009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-podinfo\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.161009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160849    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:03.161009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-certs\") pod \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\" (UID: \"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5\") "
Jul 24 13:42:03.161009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:03.161009 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.160994    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:03.161337 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.161204    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:03.161337 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.161325    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.161440 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.161347    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.161494 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.161416    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.161548 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.161499    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.161729 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.161679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:03.163028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.162997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:03.163146 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.163083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:03.163146 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.163136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:42:03.163227 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.163137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-kube-api-access-swhvq" (OuterVolumeSpecName: "kube-api-access-swhvq") pod "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" (UID: "ef8b9e3a-c8a6-482a-9592-9344ace6ffc5"). InnerVolumeSpecName "kube-api-access-swhvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:03.262537 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.262508    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.262537 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.262530    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.262537 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.262538    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.262714 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.262546    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swhvq\" (UniqueName: \"kubernetes.io/projected/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-kube-api-access-swhvq\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.262714 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.262555    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:03.599644 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.599571    2579 generic.go:356] "Generic (PLEG): container finished" podID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerID="453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82" exitCode=0
Jul 24 13:42:03.599644 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.599622    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"
Jul 24 13:42:03.599831 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.599642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" event={"ID":"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5","Type":"ContainerDied","Data":"453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82"}
Jul 24 13:42:03.599831 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.599682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc" event={"ID":"ef8b9e3a-c8a6-482a-9592-9344ace6ffc5","Type":"ContainerDied","Data":"a1e66e2493010e708487057bf6f62767badb8f90f8153d3e3f11392e9f758503"}
Jul 24 13:42:03.599831 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.599703    2579 scope.go:117] "RemoveContainer" containerID="453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82"
Jul 24 13:42:03.609425 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.609409    2579 scope.go:117] "RemoveContainer" containerID="e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106"
Jul 24 13:42:03.617452 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.617437    2579 scope.go:117] "RemoveContainer" containerID="453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82"
Jul 24 13:42:03.617702 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:42:03.617687    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82\": container with ID starting with 453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82 not found: ID does not exist" containerID="453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82"
Jul 24 13:42:03.617771 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.617708    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82"} err="failed to get container status \"453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82\": rpc error: code = NotFound desc = could not find container \"453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82\": container with ID starting with 453dbce1c862a1bb7e213e2d9563718c10fa133a8aba1207df2f34d3331f5e82 not found: ID does not exist"
Jul 24 13:42:03.617771 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.617722    2579 scope.go:117] "RemoveContainer" containerID="e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106"
Jul 24 13:42:03.617956 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:42:03.617917    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106\": container with ID starting with e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106 not found: ID does not exist" containerID="e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106"
Jul 24 13:42:03.618010 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.617950    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106"} err="failed to get container status \"e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106\": rpc error: code = NotFound desc = could not find container \"e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106\": container with ID starting with e824e023eb46cbffbf99676677c7b47a9274238f3363bc88e52fff707e207106 not found: ID does not exist"
Jul 24 13:42:03.623623 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.623603    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"]
Jul 24 13:42:03.628586 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:03.628566    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-f3a913-openshift-default-768b5c9d8-flfjc"]
Jul 24 13:42:04.563690 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:04.563655    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" path="/var/lib/kubelet/pods/ef8b9e3a-c8a6-482a-9592-9344ace6ffc5/volumes"
Jul 24 13:42:08.793676 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:08.793645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"]
Jul 24 13:42:08.794152 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:08.793905    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl" podUID="6c27e23f-fa01-402e-b276-d0a9eb35d967" containerName="maas-api" containerID="cri-o://46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10" gracePeriod=30
Jul 24 13:42:09.045462 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.045338    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:42:09.112583 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.112552    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c27e23f-fa01-402e-b276-d0a9eb35d967-openshift-service-ca\") pod \"6c27e23f-fa01-402e-b276-d0a9eb35d967\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") "
Jul 24 13:42:09.112583 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.112583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls\") pod \"6c27e23f-fa01-402e-b276-d0a9eb35d967\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") "
Jul 24 13:42:09.112803 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.112631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlddb\" (UniqueName: \"kubernetes.io/projected/6c27e23f-fa01-402e-b276-d0a9eb35d967-kube-api-access-dlddb\") pod \"6c27e23f-fa01-402e-b276-d0a9eb35d967\" (UID: \"6c27e23f-fa01-402e-b276-d0a9eb35d967\") "
Jul 24 13:42:09.113003 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.112978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c27e23f-fa01-402e-b276-d0a9eb35d967-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c27e23f-fa01-402e-b276-d0a9eb35d967" (UID: "6c27e23f-fa01-402e-b276-d0a9eb35d967"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:09.114765 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.114737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c27e23f-fa01-402e-b276-d0a9eb35d967-kube-api-access-dlddb" (OuterVolumeSpecName: "kube-api-access-dlddb") pod "6c27e23f-fa01-402e-b276-d0a9eb35d967" (UID: "6c27e23f-fa01-402e-b276-d0a9eb35d967"). InnerVolumeSpecName "kube-api-access-dlddb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:09.114862 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.114741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c27e23f-fa01-402e-b276-d0a9eb35d967" (UID: "6c27e23f-fa01-402e-b276-d0a9eb35d967"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:42:09.213392 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.213359    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c27e23f-fa01-402e-b276-d0a9eb35d967-openshift-service-ca\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:09.213392 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.213386    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c27e23f-fa01-402e-b276-d0a9eb35d967-maas-api-tls\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:09.213392 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.213396    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlddb\" (UniqueName: \"kubernetes.io/projected/6c27e23f-fa01-402e-b276-d0a9eb35d967-kube-api-access-dlddb\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:09.640238 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.640205    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c27e23f-fa01-402e-b276-d0a9eb35d967" containerID="46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10" exitCode=0
Jul 24 13:42:09.640388 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.640248    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"
Jul 24 13:42:09.640388 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.640295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl" event={"ID":"6c27e23f-fa01-402e-b276-d0a9eb35d967","Type":"ContainerDied","Data":"46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10"}
Jul 24 13:42:09.640388 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.640344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl" event={"ID":"6c27e23f-fa01-402e-b276-d0a9eb35d967","Type":"ContainerDied","Data":"14d725d1a0d2539a63779614fb78f6e94cc9a8eac318bf796f413bbd8ec1dfc8"}
Jul 24 13:42:09.640388 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.640367    2579 scope.go:117] "RemoveContainer" containerID="46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10"
Jul 24 13:42:09.650450 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.650433    2579 scope.go:117] "RemoveContainer" containerID="46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10"
Jul 24 13:42:09.650679 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:42:09.650662    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10\": container with ID starting with 46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10 not found: ID does not exist" containerID="46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10"
Jul 24 13:42:09.650717 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.650688    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10"} err="failed to get container status \"46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10\": rpc error: code = NotFound desc = could not find container \"46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10\": container with ID starting with 46da538e8c498aaee56689aa21ecb92559aaadaaa9b20ed189f46e5560002e10 not found: ID does not exist"
Jul 24 13:42:09.665471 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.665451    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"]
Jul 24 13:42:09.667518 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:09.667499    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1a3df-54fb58754-569jl"]
Jul 24 13:42:10.563753 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:10.563716    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c27e23f-fa01-402e-b276-d0a9eb35d967" path="/var/lib/kubelet/pods/6c27e23f-fa01-402e-b276-d0a9eb35d967/volumes"
Jul 24 13:42:21.855481 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:21.855391    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"]
Jul 24 13:42:21.855906 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:21.855639    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy" containerID="cri-o://b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de" gracePeriod=30
Jul 24 13:42:23.772252 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:23.772219    2579 patch_prober.go:28] interesting pod/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 13:42:23.772597 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:23.772277    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 13:42:27.104231 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.104208    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:42:27.174614 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-envoy\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.174786 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174642    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78d5r\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-kube-api-access-78d5r\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.174786 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-token\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.174786 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-credential-socket\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.174786 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-podinfo\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.175020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174792    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-certs\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.175020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174817    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-data\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.175020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-socket\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.175020 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.174918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istiod-ca-cert\") pod \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\" (UID: \"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8\") "
Jul 24 13:42:27.175216 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.175051    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:27.175348 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.175326    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:27.175551 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.175499    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-credential-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.175551 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.175516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:27.175900 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.175871    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:27.176028 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.175965    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:27.177074 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.177046    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:42:27.177074 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.177063    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:27.177192 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.177129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 13:42:27.177192 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.177144    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-kube-api-access-78d5r" (OuterVolumeSpecName: "kube-api-access-78d5r") pod "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" (UID: "c0c7e77a-3ae1-497c-98de-62f2e5b4bda8"). InnerVolumeSpecName "kube-api-access-78d5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:27.276311 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.276285    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-socket\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.276311 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.276310    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istiod-ca-cert\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.276448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.276319    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-envoy\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.276448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.276329    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78d5r\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-kube-api-access-78d5r\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.276448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.276338    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-token\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.276448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.276345    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-podinfo\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.276448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.276352    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-workload-certs\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.276448 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.276361    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8-istio-data\") on node \"ip-10-0-141-29.ec2.internal\" DevicePath \"\""
Jul 24 13:42:27.737435 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.737401    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerID="b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de" exitCode=0
Jul 24 13:42:27.737631 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.737459    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"
Jul 24 13:42:27.737631 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.737474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" event={"ID":"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8","Type":"ContainerDied","Data":"b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de"}
Jul 24 13:42:27.737631 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.737528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht" event={"ID":"c0c7e77a-3ae1-497c-98de-62f2e5b4bda8","Type":"ContainerDied","Data":"ff23d416f63e39011495d36df199848751d50798c4116a3d9bc746f0eca49f87"}
Jul 24 13:42:27.737631 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.737550    2579 scope.go:117] "RemoveContainer" containerID="b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de"
Jul 24 13:42:27.746776 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.746757    2579 scope.go:117] "RemoveContainer" containerID="707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1"
Jul 24 13:42:27.755071 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.755055    2579 scope.go:117] "RemoveContainer" containerID="b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de"
Jul 24 13:42:27.755314 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:42:27.755294    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de\": container with ID starting with b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de not found: ID does not exist" containerID="b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de"
Jul 24 13:42:27.755364 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.755323    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de"} err="failed to get container status \"b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de\": rpc error: code = NotFound desc = could not find container \"b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de\": container with ID starting with b03642bc81c257abff9f8dfe2489f1d4bde7bd0583f0dd6d954549b8eab7b8de not found: ID does not exist"
Jul 24 13:42:27.755364 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.755339    2579 scope.go:117] "RemoveContainer" containerID="707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1"
Jul 24 13:42:27.755586 ip-10-0-141-29 kubenswrapper[2579]: E0724 13:42:27.755569    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1\": container with ID starting with 707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1 not found: ID does not exist" containerID="707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1"
Jul 24 13:42:27.755643 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.755591    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1"} err="failed to get container status \"707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1\": rpc error: code = NotFound desc = could not find container \"707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1\": container with ID starting with 707d11714613db1d6951d9f07c744e8887db4f5f80f244744f39cf32822959a1 not found: ID does not exist"
Jul 24 13:42:27.763165 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.763145    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"]
Jul 24 13:42:27.767868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:27.767845    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b1a3df-openshift-default-c786c6f7b-bn6ht"]
Jul 24 13:42:28.290054 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:28.290019    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-595b47dc5f-7ltrv_ae66f370-8468-49d2-95c3-c480312dd778/manager/0.log"
Jul 24 13:42:28.521166 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:28.521126    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gj9rt_57112a2b-02a9-43e2-9fa1-e717c92a4798/manager/0.log"
Jul 24 13:42:28.563425 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:28.563355    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" path="/var/lib/kubelet/pods/c0c7e77a-3ae1-497c-98de-62f2e5b4bda8/volumes"
Jul 24 13:42:28.644539 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:28.644509    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bd76cdcf-99vcd_a1292528-bf0c-44b9-84d7-a8c29bc244fe/manager/0.log"
Jul 24 13:42:28.653207 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:28.653178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bd76cdcf-99vcd_a1292528-bf0c-44b9-84d7-a8c29bc244fe/copy-manifests/0.log"
Jul 24 13:42:29.389286 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:29.389258    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748315-wffxm_f11444f1-8ab4-4308-9db0-812d803c4ef9/cleanup/0.log"
Jul 24 13:42:30.409014 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.408989    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq_e8d7eae2-427a-44d2-989d-7f357e36101d/extract/0.log"
Jul 24 13:42:30.416337 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.416316    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq_e8d7eae2-427a-44d2-989d-7f357e36101d/util/0.log"
Jul 24 13:42:30.423603 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.423587    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq_e8d7eae2-427a-44d2-989d-7f357e36101d/pull/0.log"
Jul 24 13:42:30.535470 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.535443    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj_17fdd71d-ae33-4712-a8a7-3aab60aa15d4/util/0.log"
Jul 24 13:42:30.542584 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.542552    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj_17fdd71d-ae33-4712-a8a7-3aab60aa15d4/pull/0.log"
Jul 24 13:42:30.549606 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.549588    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj_17fdd71d-ae33-4712-a8a7-3aab60aa15d4/extract/0.log"
Jul 24 13:42:30.667259 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.667179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl_4282687f-1d59-4141-ba82-e597b9885c80/util/0.log"
Jul 24 13:42:30.677266 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.677241    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl_4282687f-1d59-4141-ba82-e597b9885c80/pull/0.log"
Jul 24 13:42:30.689216 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.689195    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl_4282687f-1d59-4141-ba82-e597b9885c80/extract/0.log"
Jul 24 13:42:30.813938 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.813902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp_33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4/util/0.log"
Jul 24 13:42:30.823279 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.823253    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp_33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4/pull/0.log"
Jul 24 13:42:30.831653 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:30.831629    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp_33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4/extract/0.log"
Jul 24 13:42:31.071367 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:31.071266    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-78qz2_1b399331-e78b-4b11-ba9c-cf460defbacf/manager/0.log"
Jul 24 13:42:31.187946 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:31.187909    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-w46b8_11abd561-d41e-48ae-b913-cce05671a8d4/manager/0.log"
Jul 24 13:42:31.302100 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:31.302074    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-z2gjj_0f5fef98-a7fd-4e52-b282-4329ace17916/kuadrant-console-plugin/0.log"
Jul 24 13:42:31.423487 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:31.423462    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7fbm6_0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8/registry-server/0.log"
Jul 24 13:42:31.546787 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:31.546761    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k4vdc_e6e9d4b5-e091-4969-9af5-e58ee89540fa/manager/0.log"
Jul 24 13:42:32.157658 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:32.157626    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w_05c7f37a-2a45-4cfa-941b-dc0ea0721541/istio-proxy/0.log"
Jul 24 13:42:32.656206 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:32.656179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkr97_8797c594-88cf-45b9-8ac9-2028500f389d/istio-proxy/1.log"
Jul 24 13:42:33.611662 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:33.611632    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rd2gf_15855414-ee67-4c02-a7be-8ebb109a07e1/main/0.log"
Jul 24 13:42:39.709456 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709424    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qx26j/must-gather-ddrjf"]
Jul 24 13:42:39.709809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709768    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy"
Jul 24 13:42:39.709809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709779    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy"
Jul 24 13:42:39.709809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709787    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy"
Jul 24 13:42:39.709809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709792    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy"
Jul 24 13:42:39.709809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709803    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy"
Jul 24 13:42:39.709809 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709809    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709814    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8fbe7c-4288-431e-bd01-bdd54bf12d56" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709819    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8fbe7c-4288-431e-bd01-bdd54bf12d56" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709825    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709830    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709836    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709841    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709849    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709853    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709860    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709865    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709870    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709875    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709881    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c27e23f-fa01-402e-b276-d0a9eb35d967" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709885    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c27e23f-fa01-402e-b276-d0a9eb35d967" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709890    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b369ab-ac94-4642-862d-23c8bbb1b1c2" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709894    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b369ab-ac94-4642-862d-23c8bbb1b1c2" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709903    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671d15b6-39d4-4c5a-b4c7-86ceff680bda" containerName="main"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709907    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="671d15b6-39d4-4c5a-b4c7-86ceff680bda" containerName="main"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709983    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709988    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8fbe7c-4288-431e-bd01-bdd54bf12d56" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709994    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.709999    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d93e51-5a27-4306-b6c5-c06f0f682027" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.710004    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c27e23f-fa01-402e-b276-d0a9eb35d967" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.710007    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.710012    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="68797318-58fd-41d1-9b61-77930b4dcd47" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.710017    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0c7e77a-3ae1-497c-98de-62f2e5b4bda8" containerName="istio-proxy"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.710024    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b369ab-ac94-4642-862d-23c8bbb1b1c2" containerName="maas-api"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.710032    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="671d15b6-39d4-4c5a-b4c7-86ceff680bda" containerName="main"
Jul 24 13:42:39.710121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.710039    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8b9e3a-c8a6-482a-9592-9344ace6ffc5" containerName="istio-proxy"
Jul 24 13:42:39.713084 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.713067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qx26j/must-gather-ddrjf"
Jul 24 13:42:39.715663 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.715641    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qx26j\"/\"kube-root-ca.crt\""
Jul 24 13:42:39.715810 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.715697    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qx26j\"/\"openshift-service-ca.crt\""
Jul 24 13:42:39.716620 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.716604    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qx26j\"/\"default-dockercfg-mg26v\""
Jul 24 13:42:39.723205 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.723184    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qx26j/must-gather-ddrjf"]
Jul 24 13:42:39.790490 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.790461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e227f56f-3c62-4c20-87f3-0d1c99fe9d1b-must-gather-output\") pod \"must-gather-ddrjf\" (UID: \"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b\") " pod="openshift-must-gather-qx26j/must-gather-ddrjf"
Jul 24 13:42:39.790615 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.790498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgpcd\" (UniqueName: \"kubernetes.io/projected/e227f56f-3c62-4c20-87f3-0d1c99fe9d1b-kube-api-access-bgpcd\") pod \"must-gather-ddrjf\" (UID: \"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b\") " pod="openshift-must-gather-qx26j/must-gather-ddrjf"
Jul 24 13:42:39.891514 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.891480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e227f56f-3c62-4c20-87f3-0d1c99fe9d1b-must-gather-output\") pod \"must-gather-ddrjf\" (UID: \"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b\") " pod="openshift-must-gather-qx26j/must-gather-ddrjf"
Jul 24 13:42:39.891678 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.891521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpcd\" (UniqueName: \"kubernetes.io/projected/e227f56f-3c62-4c20-87f3-0d1c99fe9d1b-kube-api-access-bgpcd\") pod \"must-gather-ddrjf\" (UID: \"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b\") " pod="openshift-must-gather-qx26j/must-gather-ddrjf"
Jul 24 13:42:39.891810 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.891793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e227f56f-3c62-4c20-87f3-0d1c99fe9d1b-must-gather-output\") pod \"must-gather-ddrjf\" (UID: \"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b\") " pod="openshift-must-gather-qx26j/must-gather-ddrjf"
Jul 24 13:42:39.902630 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:39.902607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpcd\" (UniqueName: \"kubernetes.io/projected/e227f56f-3c62-4c20-87f3-0d1c99fe9d1b-kube-api-access-bgpcd\") pod \"must-gather-ddrjf\" (UID: \"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b\") " pod="openshift-must-gather-qx26j/must-gather-ddrjf"
Jul 24 13:42:40.021918 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:40.021842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qx26j/must-gather-ddrjf"
Jul 24 13:42:40.157164 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:40.157138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qx26j/must-gather-ddrjf"]
Jul 24 13:42:40.159344 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:42:40.159315    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode227f56f_3c62_4c20_87f3_0d1c99fe9d1b.slice/crio-d10aea80132e0e38084042fb330fc30821c2df3bf08c63aadf9440756aed21be WatchSource:0}: Error finding container d10aea80132e0e38084042fb330fc30821c2df3bf08c63aadf9440756aed21be: Status 404 returned error can't find the container with id d10aea80132e0e38084042fb330fc30821c2df3bf08c63aadf9440756aed21be
Jul 24 13:42:40.810604 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:40.810569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qx26j/must-gather-ddrjf" event={"ID":"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b","Type":"ContainerStarted","Data":"d10aea80132e0e38084042fb330fc30821c2df3bf08c63aadf9440756aed21be"}
Jul 24 13:42:41.821609 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:41.821566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qx26j/must-gather-ddrjf" event={"ID":"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b","Type":"ContainerStarted","Data":"e297128459b1beca65ed798d74fc0a38eab469847b3e4366f4aa4661af1da6a5"}
Jul 24 13:42:41.822052 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:41.821621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qx26j/must-gather-ddrjf" event={"ID":"e227f56f-3c62-4c20-87f3-0d1c99fe9d1b","Type":"ContainerStarted","Data":"5554ddd611d612073422e3b647e9fcf40cb1b757b06a92266e10b717c736e789"}
Jul 24 13:42:41.842934 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:41.842864    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qx26j/must-gather-ddrjf" podStartSLOduration=2.102101496 podStartE2EDuration="2.842844083s" totalImagesPullingTime="740.742587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:41.841019354 +0000 UTC m=+2985.830338049" watchObservedRunningTime="2026-07-24 13:42:41.842844083 +0000 UTC m=+2985.832162781"
Jul 24 13:42:42.622551 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:42.622520    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-shckb_6956353c-c03e-4a3c-8a33-a098f4c11e35/global-pull-secret-syncer/0.log"
Jul 24 13:42:42.767511 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:42.767482    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-stggw_b3888e8a-388c-496d-bdbc-8f266fb32067/konnectivity-agent/0.log"
Jul 24 13:42:42.866807 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:42.866774    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-29.ec2.internal_630256be83544d972c8b0ff554f26b54/haproxy/0.log"
Jul 24 13:42:46.883594 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:46.883565    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq_e8d7eae2-427a-44d2-989d-7f357e36101d/extract/0.log"
Jul 24 13:42:46.922833 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:46.922808    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq_e8d7eae2-427a-44d2-989d-7f357e36101d/util/0.log"
Jul 24 13:42:46.959130 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:46.959078    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z74hq_e8d7eae2-427a-44d2-989d-7f357e36101d/pull/0.log"
Jul 24 13:42:46.991360 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:46.991290    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj_17fdd71d-ae33-4712-a8a7-3aab60aa15d4/extract/0.log"
Jul 24 13:42:47.020167 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.020140    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj_17fdd71d-ae33-4712-a8a7-3aab60aa15d4/util/0.log"
Jul 24 13:42:47.057057 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.057030    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7prj_17fdd71d-ae33-4712-a8a7-3aab60aa15d4/pull/0.log"
Jul 24 13:42:47.103990 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.103962    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl_4282687f-1d59-4141-ba82-e597b9885c80/extract/0.log"
Jul 24 13:42:47.135527 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.135451    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl_4282687f-1d59-4141-ba82-e597b9885c80/util/0.log"
Jul 24 13:42:47.171947 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.171905    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h6jfl_4282687f-1d59-4141-ba82-e597b9885c80/pull/0.log"
Jul 24 13:42:47.206692 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.206655    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp_33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4/extract/0.log"
Jul 24 13:42:47.242917 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.242885    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp_33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4/util/0.log"
Jul 24 13:42:47.276135 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.276078    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vvrmp_33b88c58-6b97-4a6f-82ac-9eb37ce9b5f4/pull/0.log"
Jul 24 13:42:47.630615 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.630582    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-78qz2_1b399331-e78b-4b11-ba9c-cf460defbacf/manager/0.log"
Jul 24 13:42:47.685575 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.685549    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-w46b8_11abd561-d41e-48ae-b913-cce05671a8d4/manager/0.log"
Jul 24 13:42:47.717244 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.717218    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-z2gjj_0f5fef98-a7fd-4e52-b282-4329ace17916/kuadrant-console-plugin/0.log"
Jul 24 13:42:47.773777 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.773749    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7fbm6_0fd7bca9-3e8e-4148-a1fe-4d1d1a5d07d8/registry-server/0.log"
Jul 24 13:42:47.944048 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:47.944013    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k4vdc_e6e9d4b5-e091-4969-9af5-e58ee89540fa/manager/0.log"
Jul 24 13:42:49.623731 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:49.623651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_537d218b-c2da-43bf-bb18-e641c0d7e26c/alertmanager/0.log"
Jul 24 13:42:49.660419 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:49.660389    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_537d218b-c2da-43bf-bb18-e641c0d7e26c/config-reloader/0.log"
Jul 24 13:42:49.701468 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:49.701311    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_537d218b-c2da-43bf-bb18-e641c0d7e26c/kube-rbac-proxy-web/0.log"
Jul 24 13:42:49.733609 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:49.733584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_537d218b-c2da-43bf-bb18-e641c0d7e26c/kube-rbac-proxy/0.log"
Jul 24 13:42:49.770733 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:49.770692    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_537d218b-c2da-43bf-bb18-e641c0d7e26c/kube-rbac-proxy-metric/0.log"
Jul 24 13:42:49.801730 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:49.801665    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_537d218b-c2da-43bf-bb18-e641c0d7e26c/prom-label-proxy/0.log"
Jul 24 13:42:49.833161 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:49.833129    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_537d218b-c2da-43bf-bb18-e641c0d7e26c/init-config-reloader/0.log"
Jul 24 13:42:50.220306 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:50.220274    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qz7x7_7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9/node-exporter/0.log"
Jul 24 13:42:50.248132 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:50.248101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qz7x7_7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9/kube-rbac-proxy/0.log"
Jul 24 13:42:50.275119 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:50.275092    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qz7x7_7788ab9f-b4bd-4b7b-87f4-a1b6b6f8e9b9/init-textfile/0.log"
Jul 24 13:42:50.728746 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:50.728712    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-8k4kq_a6a3a4eb-576a-48d5-97e8-06f233fdca9a/prometheus-operator/0.log"
Jul 24 13:42:50.752107 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:50.752075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-8k4kq_a6a3a4eb-576a-48d5-97e8-06f233fdca9a/kube-rbac-proxy/0.log"
Jul 24 13:42:50.781695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:50.781659    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-vwcg8_f4caef4b-fd87-4760-b8d8-3957496cc44e/prometheus-operator-admission-webhook/0.log"
Jul 24 13:42:52.072526 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.072489    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"]
Jul 24 13:42:52.078554 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.078526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.083868 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.083815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"]
Jul 24 13:42:52.226121 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.226090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-podres\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.226315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.226171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdfg\" (UniqueName: \"kubernetes.io/projected/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-kube-api-access-kcdfg\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.226315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.226197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-proc\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.226315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.226261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-lib-modules\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.226315 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.226292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-sys\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327052 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.326955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdfg\" (UniqueName: \"kubernetes.io/projected/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-kube-api-access-kcdfg\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327052 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.327014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-proc\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327287 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.327069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-lib-modules\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327287 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.327102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-sys\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327287 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.327167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-podres\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327440 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.327347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-podres\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327735 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.327626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-lib-modules\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327735 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.327700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-proc\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.327735 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.327702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-sys\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.337070 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.337042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdfg\" (UniqueName: \"kubernetes.io/projected/b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c-kube-api-access-kcdfg\") pod \"perf-node-gather-daemonset-l4tvr\" (UID: \"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.366057 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.366022    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-twrff_f579dc6a-bffa-499c-8540-0e952bb62a06/networking-console-plugin/0.log"
Jul 24 13:42:52.395502 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.395310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.584306 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.584279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"]
Jul 24 13:42:52.586567 ip-10-0-141-29 kubenswrapper[2579]: W0724 13:42:52.586541    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb2a2bff2_e4b7_42e0_a8d4_e9fa3c0d119c.slice/crio-5be5655abf0772d35d7b86bf6f48256bb232dfabc5113c635b9394dac4885a15 WatchSource:0}: Error finding container 5be5655abf0772d35d7b86bf6f48256bb232dfabc5113c635b9394dac4885a15: Status 404 returned error can't find the container with id 5be5655abf0772d35d7b86bf6f48256bb232dfabc5113c635b9394dac4885a15
Jul 24 13:42:52.923683 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.923338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr" event={"ID":"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c","Type":"ContainerStarted","Data":"cdec565888bb8de98539628a4e09661a5ef085ad0732ea633cc73a78a61bed55"}
Jul 24 13:42:52.923683 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.923438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr" event={"ID":"b2a2bff2-e4b7-42e0-a8d4-e9fa3c0d119c","Type":"ContainerStarted","Data":"5be5655abf0772d35d7b86bf6f48256bb232dfabc5113c635b9394dac4885a15"}
Jul 24 13:42:52.923683 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.923643    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:52.943223 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:52.943162    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr" podStartSLOduration=0.943141784 podStartE2EDuration="943.141784ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:52.940391635 +0000 UTC m=+2996.929710330" watchObservedRunningTime="2026-07-24 13:42:52.943141784 +0000 UTC m=+2996.932460486"
Jul 24 13:42:53.584356 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:53.584327    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7f5c8d5d86-khztv_48b225b2-12e6-4e6d-a3fa-b79be6875bfc/console/0.log"
Jul 24 13:42:53.624618 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:53.624587    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-wrsk5_ab5c680f-75a8-4c64-a9cc-70ce761974dd/download-server/0.log"
Jul 24 13:42:55.297185 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:55.297150    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hp2hl_8354fa4f-cf8e-479d-bfc2-59514dff4b1c/dns/0.log"
Jul 24 13:42:55.322545 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:55.322524    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hp2hl_8354fa4f-cf8e-479d-bfc2-59514dff4b1c/kube-rbac-proxy/0.log"
Jul 24 13:42:55.480201 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:55.480175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qf5nm_56249eea-b34a-4073-968c-a9e93e64e999/dns-node-resolver/0.log"
Jul 24 13:42:56.087710 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:56.087671    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-9d76c648c-5m9xs_7cae0063-69fe-4e37-a7cd-cbf5be03f541/registry/0.log"
Jul 24 13:42:56.168398 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:56.168369    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zl565_a39d5644-41ba-439f-87e2-72f2dfae3422/node-ca/0.log"
Jul 24 13:42:57.165480 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:57.165443    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w_05c7f37a-2a45-4cfa-941b-dc0ea0721541/istio-proxy/0.log"
Jul 24 13:42:57.520143 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:57.520114    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkr97_8797c594-88cf-45b9-8ac9-2028500f389d/istio-proxy/0.log"
Jul 24 13:42:57.556173 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:57.556140    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkr97_8797c594-88cf-45b9-8ac9-2028500f389d/istio-proxy/1.log"
Jul 24 13:42:58.453883 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:58.453854    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8mj9k_c42f561d-917e-4912-9c0d-7c7adcbdfb86/serve-healthcheck-canary/0.log"
Jul 24 13:42:58.942956 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:58.942895    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-l4tvr"
Jul 24 13:42:59.063499 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:59.063469    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmg69_e2ff83e6-089c-4920-ae44-4f4c8fbc5737/kube-rbac-proxy/0.log"
Jul 24 13:42:59.096768 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:59.096741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmg69_e2ff83e6-089c-4920-ae44-4f4c8fbc5737/exporter/0.log"
Jul 24 13:42:59.135137 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:42:59.135099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmg69_e2ff83e6-089c-4920-ae44-4f4c8fbc5737/extractor/0.log"
Jul 24 13:43:02.187661 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:02.187604    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w_05c7f37a-2a45-4cfa-941b-dc0ea0721541/istio-proxy/0.log"
Jul 24 13:43:02.289509 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:02.289474    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-595b47dc5f-7ltrv_ae66f370-8468-49d2-95c3-c480312dd778/manager/0.log"
Jul 24 13:43:02.413003 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:02.412975    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gj9rt_57112a2b-02a9-43e2-9fa1-e717c92a4798/manager/0.log"
Jul 24 13:43:02.457576 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:02.457552    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bd76cdcf-99vcd_a1292528-bf0c-44b9-84d7-a8c29bc244fe/manager/0.log"
Jul 24 13:43:02.490000 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:02.489981    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bd76cdcf-99vcd_a1292528-bf0c-44b9-84d7-a8c29bc244fe/copy-manifests/0.log"
Jul 24 13:43:02.518835 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:02.518812    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkr97_8797c594-88cf-45b9-8ac9-2028500f389d/istio-proxy/0.log"
Jul 24 13:43:02.548090 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:02.548066    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkr97_8797c594-88cf-45b9-8ac9-2028500f389d/istio-proxy/1.log"
Jul 24 13:43:04.056147 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:04.056118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-756cc79bf9-hpgrp_e55e2688-3ee0-4915-9bcf-14b4e683248a/manager/0.log"
Jul 24 13:43:04.098376 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:04.098340    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nb72w_05c7f37a-2a45-4cfa-941b-dc0ea0721541/istio-proxy/0.log"
Jul 24 13:43:04.120086 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:04.120056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gd5tn_6c8fb660-7f39-4724-b1d4-b89e57bcf4c7/openshift-lws-operator/0.log"
Jul 24 13:43:04.435695 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:04.435664    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkr97_8797c594-88cf-45b9-8ac9-2028500f389d/istio-proxy/0.log"
Jul 24 13:43:04.468222 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:04.468192    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkr97_8797c594-88cf-45b9-8ac9-2028500f389d/istio-proxy/1.log"
Jul 24 13:43:09.222704 ip-10-0-141-29 kubenswrapper[2579]: I0724 13:43:09.222666    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-crg9r_e336ec07-ea3f-45ac-9502-d55d3803b42c/migrator/0.log"