Sep 11 21:52:18.222915 ip-10-0-141-45 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 21:52:18.607167 ip-10-0-141-45 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:18.607167 ip-10-0-141-45 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 21:52:18.607167 ip-10-0-141-45 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:18.607167 ip-10-0-141-45 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 21:52:18.607167 ip-10-0-141-45 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:18.608336 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.608259    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 21:52:18.616652 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616621    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:18.616652 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616648    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:18.616652 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616653    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616656    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616660    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616662    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616665    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616668    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616671    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616681    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616685    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616688    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616690    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616694    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616699    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616703    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616706    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616709    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616711    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616714    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616717    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:18.616756 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616720    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616722    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616725    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616729    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616732    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616735    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616738    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616741    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616743    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616746    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616748    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616751    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616753    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616756    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616758    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616761    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616764    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616767    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616770    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616773    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:18.617229 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616777    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616780    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616782    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616785    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616788    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616790    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616792    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616797    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616800    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616803    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616806    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616808    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616811    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616813    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616817    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616819    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616822    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616825    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616827    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616830    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:18.618086 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616833    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616835    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616838    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616840    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616843    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616845    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616848    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616850    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616853    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616856    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616859    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616861    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616865    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616867    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616870    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616872    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616875    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616878    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616880    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616883    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:18.618689 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616885    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616888    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616890    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616893    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616896    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616899    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616902    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616905    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616907    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616910    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616913    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616915    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616918    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616920    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616923    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616925    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616928    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:18.619193 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.616930    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:18.619608 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619585    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:18.619608 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619598    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:18.619608 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619603    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:18.619608 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619607    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:18.619608 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619611    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619614    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619617    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619620    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619623    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619626    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619629    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619648    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619651    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619654    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619657    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619660    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619662    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619665    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619668    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619671    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619673    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619676    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619686    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619689    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:18.619752 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619692    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619694    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619697    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619700    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619702    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619705    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619707    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619709    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619712    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619715    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619717    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619720    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619722    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619725    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619728    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619731    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619733    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619736    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619739    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619741    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:18.620257 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619744    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619746    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619749    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619751    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619754    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619758    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619763    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619765    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619768    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619771    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619774    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619776    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619779    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619781    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619784    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619786    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619789    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619792    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619794    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619797    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:18.620803 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619800    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619802    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619807    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619812    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619815    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619818    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619821    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619824    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619827    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619830    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619833    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619835    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619838    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619841    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619843    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619846    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619848    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619851    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619853    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:18.621293 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619862    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619865    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619867    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619870    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619872    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619875    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619877    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619880    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619883    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619885    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619887    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619890    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619893    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619895    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619898    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.619900    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620307    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620312    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620315    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620318    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:18.621788 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620321    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620324    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620326    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620329    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620332    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620334    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620337    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620339    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620342    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620345    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620348    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620350    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620359    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620362    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620364    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620367    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620369    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620372    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620376    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:18.622283 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620381    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620384    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620387    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620389    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620392    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620394    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620397    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620399    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620402    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620404    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620407    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620410    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620414    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620416    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620419    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620421    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620424    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620426    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620429    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620431    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:18.622784 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620434    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620438    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620441    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620445    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620448    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620451    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620456    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620459    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620462    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620464    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620467    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620469    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620472    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620475    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620478    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620480    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620483    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620486    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620489    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620491    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:18.623278 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620494    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620496    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620499    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620502    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620504    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620507    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620509    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620512    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620514    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620517    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620519    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620522    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620525    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620527    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620530    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620533    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620535    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620539    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620542    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:18.623796 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620546    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620549    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620552    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620555    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620558    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620562    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620566    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620568    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620571    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620574    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620577    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620580    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620582    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620585    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620588    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620590    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.620593    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620682    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620727    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620736    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620740    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 21:52:18.624264 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620745    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620748    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620752    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620759    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620762    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620765    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620769    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620772    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620775    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620778    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620781    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620784    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620788    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620791    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620795    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620799    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620802    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620805    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620809    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620813    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620816    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620819    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620823    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620826    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 21:52:18.624797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620829    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620832    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620835    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620838    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620842    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620845    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620848    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620851    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620854    2578 flags.go:64] FLAG: --enable-server="true"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620857    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620860    2578 flags.go:64] FLAG: --event-burst="100"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620863    2578 flags.go:64] FLAG: --event-qps="50"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620866    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620869    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620872    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620876    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620879    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620882    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620885    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620887    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620890    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620894    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620897    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620900    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620905    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 21:52:18.625402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620908    2578 flags.go:64] FLAG: --feature-gates=""
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620911    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620914    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620918    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620921    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620924    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620927    2578 flags.go:64] FLAG: --help="false"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620930    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-141-45.ec2.internal"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620933    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620936    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620939    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620943    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620946    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620949    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620952    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620955    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620958    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620961    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620964    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620967    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620969    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620972    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620975    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620978    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 21:52:18.626056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620981    2578 flags.go:64] FLAG: --lock-file=""
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620983    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620986    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620989    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620994    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.620998    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621001    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621005    2578 flags.go:64] FLAG: --logging-format="text"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621008    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621011    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621014    2578 flags.go:64] FLAG: --manifest-url=""
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621017    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621021    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621024    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621028    2578 flags.go:64] FLAG: --max-pods="110"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621031    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621034    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621037    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621040    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621043    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621046    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621049    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621056    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621060    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621062    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 21:52:18.626671 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621065    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621068    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621073    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621076    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621079    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621082    2578 flags.go:64] FLAG: --port="10250"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621085    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621088    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e6a04751f7f61d55"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621092    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621095    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621098    2578 flags.go:64] FLAG: --register-node="true"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621101    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621105    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621109    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621112    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621116    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621119    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621122    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621126    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621128    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621131    2578 flags.go:64] FLAG: --runonce="false"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621134    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621137    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621140    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621143    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 21:52:18.627348 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621146    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621149    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621152    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621155    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621158    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621161    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621164    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621167    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621170    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621173    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621176    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621181    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621184    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621187    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621193    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621195    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621198    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621201    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621204    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621207    2578 flags.go:64] FLAG: --v="2"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621210    2578 flags.go:64] FLAG: --version="false"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621219    2578 flags.go:64] FLAG: --vmodule=""
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621225    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.621228    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621346    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:18.627990 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621349    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621352    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621355    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621358    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621361    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621363    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621366    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621369    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621371    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621374    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621376    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621379    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621382    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621384    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621387    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621390    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621394    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621398    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621401    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:18.628597 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621404    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621407    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621411    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621414    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621416    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621419    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621421    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621425    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621428    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621430    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621434    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621437    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621439    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621442    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621444    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621447    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621449    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621452    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621454    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621457    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:18.629116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621460    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621462    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621465    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621467    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621470    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621472    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621475    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621477    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621480    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621482    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621485    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621487    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621490    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621492    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621496    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621499    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621501    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621504    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621506    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621509    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:18.629605 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621511    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621514    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621517    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621520    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621523    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621525    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621528    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621531    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621533    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621536    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621538    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621541    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621543    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621546    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621548    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621551    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621554    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621557    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621559    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621561    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:18.630151 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621564    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621567    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621569    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621572    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621574    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621577    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621582    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621586    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621590    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621594    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621597    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621601    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621605    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621609    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621613    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621617    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621621    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621624    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:18.630663 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.621626    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:18.631110 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.622296    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:18.631110 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.630064    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 21:52:18.631110 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.630080    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 21:52:18.631279 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.631268    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 21:52:18.631312 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.631279    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 21:52:18.631359 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631351    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:18.631359 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631358    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631361    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631364    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631367    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631370    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631373    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631376    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631379    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631382    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631385    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631388    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631390    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631393    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631396    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631399    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631402    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631405    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631407    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631410    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631412    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:18.631412 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631415    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631418    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631421    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631425    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631428    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631430    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631433    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631436    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631438    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631441    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631444    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631447    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631450    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631453    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631455    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631458    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631461    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631463    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631466    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631469    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:18.631975 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631472    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631474    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631477    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631482    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631485    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631489    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631492    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631495    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631498    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631501    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631503    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631506    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631508    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631511    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631515    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631518    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631521    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631525    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:18.632465 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631529    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631532    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631535    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631538    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631541    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631544    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631547    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631550    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631553    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631555    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631558    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631561    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631563    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631566    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631569    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631571    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631574    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631576    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631579    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631582    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:18.632917 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631585    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631587    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631590    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631593    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631595    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631598    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631601    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631604    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631607    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631610    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631612    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631615    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631618    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631621    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631624    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631626    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631629    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631645    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631650    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:18.633429 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631655    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.631660    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631771    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631775    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631778    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631781    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631784    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631787    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631790    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631793    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631796    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631799    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631801    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631805    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:18.633914 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631807    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631810    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631812    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631815    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631817    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631820    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631822    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631825    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631828    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631831    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631833    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631837    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631839    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631842    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631845    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631848    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631850    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631853    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631855    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631859    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:18.634271 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631862    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631865    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631868    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631871    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631874    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631876    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631880    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631884    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631887    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631890    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631893    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631896    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631899    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631902    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631904    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631907    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631910    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631913    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631916    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:18.634782 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631918    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631921    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631923    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631926    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631929    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631932    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631934    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631937    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631939    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631942    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631945    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631947    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631949    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631952    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631955    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631957    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631960    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631962    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631965    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:18.635275 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631967    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631970    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631972    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631975    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631977    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631980    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631983    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631986    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631989    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631991    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631994    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631997    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.631999    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632002    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632005    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632007    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632010    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632012    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632015    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632018    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:18.635840 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632020    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632023    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632025    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632028    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632030    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632033    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632036    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632038    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:18.632041    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.632046    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.632149    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 21:52:18.636329 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.635613    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 21:52:18.636626 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.636614    2578 server.go:1019] "Starting client certificate rotation"
Sep 11 21:52:18.636745 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.636728    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:52:18.636776 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.636771    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:52:18.640191 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.640178    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:52:18.643287 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.643274    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:52:18.656275 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.656255    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 11 21:52:18.659555 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.659531    2578 log.go:25] "Validated CRI v1 image API"
Sep 11 21:52:18.660443 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.660426    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 21:52:18.664357 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.664341    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:52:18.666968 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.666948    2578 fs.go:135] Filesystem UUIDs: map[6f53ff0c-607a-4162-9bf2-9d6081353f75:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d09ae5b1-01de-4890-8f6c-7d9e5095cb5d:/dev/nvme0n1p3]
Sep 11 21:52:18.667022 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.666969    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 21:52:18.672219 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.672107    2578 manager.go:216] Machine: {Timestamp:2026-09-11 21:52:18.670666959 +0000 UTC m=+0.355613664 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3130101 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229d995047f295c675e13a829a502c SystemUUID:ec229d99-5047-f295-c675-e13a829a502c BootID:f267514b-e670-47ae-a6f3-c3f31659b3d3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:5a:48:27:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:5a:48:27:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:2f:42:cf:e0:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 21:52:18.672219 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.672217    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 21:52:18.672338 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.672330    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 21:52:18.675934 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.675903    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 21:52:18.676093 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.675937    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 21:52:18.676791 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.676782    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 21:52:18.676831 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.676793    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 21:52:18.676831 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.676801    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:52:18.677694 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.677684    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:52:18.679578 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.679566    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 21:52:18.684321 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684307    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:52:18.684453 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684443    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 21:52:18.684484 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684477    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 21:52:18.684519 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684488    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 21:52:18.684519 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684503    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 21:52:18.684519 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684512    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 11 21:52:18.684606 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684522    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 21:52:18.684917 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684904    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:52:18.684917 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.684922    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:52:18.687446 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.687420    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tzcmr"
Sep 11 21:52:18.688458 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.688443    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 21:52:18.691357 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.691342    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 21:52:18.691419 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.691373    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 21:52:18.694029 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694014    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tzcmr"
Sep 11 21:52:18.694081 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.694054    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 21:52:18.694119 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.694059    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 21:52:18.694383 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694370    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 21:52:18.694414 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694394    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 21:52:18.694414 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694402    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 21:52:18.694414 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694407    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 21:52:18.694498 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694416    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 21:52:18.694498 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694422    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 21:52:18.694498 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694436    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 21:52:18.694498 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694442    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 21:52:18.694498 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694450    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 21:52:18.694498 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694456    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 21:52:18.694498 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694464    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 21:52:18.694498 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.694469    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 21:52:18.695364 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.695352    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 21:52:18.695399 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.695367    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 21:52:18.699388 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.699374    2578 server.go:1268] "Started kubelet"
Sep 11 21:52:18.699490 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.699439    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 21:52:18.699533 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.699498    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 21:52:18.704213 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.704192    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 21:52:18.704445 ip-10-0-141-45 systemd[1]: Started Kubernetes Kubelet.
Sep 11 21:52:18.708411 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.708341    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 21:52:18.708491 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.708455    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 21:52:18.709246 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.709209    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 21:52:18.715054 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.715037    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 21:52:18.715775 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.715758    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 21:52:18.715966 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.715932    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:18.716188 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716166    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-45.ec2.internal" not found
Sep 11 21:52:18.716724 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716708    2578 factory.go:55] Registering systemd factory
Sep 11 21:52:18.716800 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716727    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 11 21:52:18.716800 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716714    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 21:52:18.716800 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716716    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 21:52:18.716800 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716759    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 21:52:18.716960 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716837    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 21:52:18.716960 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716846    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 21:52:18.716960 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716930    2578 factory.go:153] Registering CRI-O factory
Sep 11 21:52:18.716960 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716944    2578 factory.go:223] Registration of the crio container factory successfully
Sep 11 21:52:18.717065 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.716990    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 21:52:18.717065 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.717006    2578 factory.go:103] Registering Raw factory
Sep 11 21:52:18.717065 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.717017    2578 manager.go:1196] Started watching for new ooms in manager
Sep 11 21:52:18.717399 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.717389    2578 manager.go:349] Starting recovery of all containers
Sep 11 21:52:18.717858 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.717841    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:18.720725 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.720705    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal"
Sep 11 21:52:18.723711 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.723664    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 21:52:18.728184 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.728165    2578 manager.go:354] Recovery completed
Sep 11 21:52:18.732907 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.732895    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:18.733767 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.733749    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-45.ec2.internal" not found
Sep 11 21:52:18.735443 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.735426    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:18.735520 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.735461    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:18.735520 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.735473    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:18.736254 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.736224    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 21:52:18.736254 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.736252    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 21:52:18.736362 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.736225    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:18.736362 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.736270    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:52:18.737730 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.737713    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:18.737792 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.737740    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:18.737792 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.737750    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:18.738487 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.738470    2578 policy_none.go:49] "None policy: Start"
Sep 11 21:52:18.738539 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.738491    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 21:52:18.738539 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.738505    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 21:52:18.740516 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.740505    2578 policy_none.go:47] "Start"
Sep 11 21:52:18.795062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.795034    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-45.ec2.internal" not found
Sep 11 21:52:18.816376 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.816346    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:18.850184 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.850146    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 21:52:18.850350 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.850223    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 21:52:18.850350 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.850254    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 21:52:18.850350 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.850302    2578 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 11 21:52:18.852779 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.852756    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:18.916770 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.916739    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:18.917308 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.917291    2578 manager.go:337] "Starting Device Plugin manager"
Sep 11 21:52:18.917400 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.917342    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 21:52:18.917400 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.917358    2578 server.go:87] "Starting device plugin registration server"
Sep 11 21:52:18.917697 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.917683    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 21:52:18.917770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.917698    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 21:52:18.917818 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.917791    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 21:52:18.917939 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.917924    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 21:52:18.917939 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.917939    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 21:52:18.918484 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.918458    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 21:52:18.918562 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.918521    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:18.950770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.950729    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"]
Sep 11 21:52:18.950924 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.950842    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:18.952450 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.952429    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:18.952573 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.952464    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:18.952573 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.952482    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:18.954904 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.954886    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:18.955038 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.955020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:18.955087 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.955061    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:18.955704 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.955688    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:18.955784 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.955719    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:18.955784 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.955733    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:18.955784 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.955692    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:18.955924 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.955794    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:18.955924 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.955805    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:18.957977 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.957954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:18.958067 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.957992    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:18.958715 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.958699    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:18.958788 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.958733    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:18.958788 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:18.958749    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:18.969222 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.969199    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal"
Sep 11 21:52:18.973696 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:18.973676    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.017797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.017768    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:19.018806 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.018763    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:19.018806 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.018804    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:19.018965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.018815    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:19.018965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.018844    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.027096 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.027076    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.027180 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.027104    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-45.ec2.internal\": node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.042003 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.041976    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.118595 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.118562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddec20afa56710be0d0517a302df8414-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"ddec20afa56710be0d0517a302df8414\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.118693 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.118596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fec26c83d95126dedd1e7e37e73c673-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"9fec26c83d95126dedd1e7e37e73c673\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.118693 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.118625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddec20afa56710be0d0517a302df8414-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"ddec20afa56710be0d0517a302df8414\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.142905 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.142847    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.219491 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.219438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddec20afa56710be0d0517a302df8414-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"ddec20afa56710be0d0517a302df8414\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.219568 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.219502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fec26c83d95126dedd1e7e37e73c673-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"9fec26c83d95126dedd1e7e37e73c673\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.219568 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.219560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddec20afa56710be0d0517a302df8414-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"ddec20afa56710be0d0517a302df8414\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.219653 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.219566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fec26c83d95126dedd1e7e37e73c673-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"9fec26c83d95126dedd1e7e37e73c673\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.219653 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.219571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddec20afa56710be0d0517a302df8414-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"ddec20afa56710be0d0517a302df8414\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.219653 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.219608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddec20afa56710be0d0517a302df8414-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"ddec20afa56710be0d0517a302df8414\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.243594 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.243561    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.272502 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.272467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.276675 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.276651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:19.343891 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.343839    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.444992 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.444900    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.546551 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.546518    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.637730 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.637695    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 21:52:19.638197 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.637863    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:52:19.638197 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.637873    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:52:19.646606 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.646580    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.700328 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.700118    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 21:47:19 +0000 UTC" deadline="2028-05-14 22:22:58.891996439 +0000 UTC"
Sep 11 21:52:19.700328 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.700303    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14664h30m39.191707027s"
Sep 11 21:52:19.747404 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.747368    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.762003 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.761971    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:19.847828 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.847797    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.893647 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:19.893604    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fec26c83d95126dedd1e7e37e73c673.slice/crio-dceb5c2f70124118ca63abd46bd2e353e15e4c9d8ccfbb635c44e76fb24466aa WatchSource:0}: Error finding container dceb5c2f70124118ca63abd46bd2e353e15e4c9d8ccfbb635c44e76fb24466aa: Status 404 returned error can't find the container with id dceb5c2f70124118ca63abd46bd2e353e15e4c9d8ccfbb635c44e76fb24466aa
Sep 11 21:52:19.894806 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:19.894782    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddec20afa56710be0d0517a302df8414.slice/crio-206bbd4b7fb9b3b4429892cdef66a39be06d9d7c8c11c88c5f503cd6e06aec4c WatchSource:0}: Error finding container 206bbd4b7fb9b3b4429892cdef66a39be06d9d7c8c11c88c5f503cd6e06aec4c: Status 404 returned error can't find the container with id 206bbd4b7fb9b3b4429892cdef66a39be06d9d7c8c11c88c5f503cd6e06aec4c
Sep 11 21:52:19.897454 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.897438    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:52:19.948407 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:19.948365    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 11 21:52:19.970901 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:19.970842    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:20.019309 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.019271    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:20.029160 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.029118    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:52:20.030262 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.030246    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 11 21:52:20.040294 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.040274    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:52:20.459045 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.458816    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:20.602887 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.602855    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:20.686126 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.686094    2578 apiserver.go:52] "Watching apiserver"
Sep 11 21:52:20.693666 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.693034    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 21:52:20.693666 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.693597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-rmt84","kube-system/global-pull-secret-syncer-cq4fq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66","openshift-cluster-node-tuning-operator/tuned-4wzr8","openshift-multus/network-metrics-daemon-992bd","openshift-network-diagnostics/network-check-target-9qqhz","openshift-ovn-kubernetes/ovnkube-node-pld4d","kube-system/konnectivity-agent-x7k89","kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal","openshift-dns/node-resolver-dj8qt","openshift-image-registry/node-ca-h8vxg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal","openshift-multus/multus-additional-cni-plugins-5ncm8","openshift-multus/multus-nrw8m"]
Sep 11 21:52:20.697127 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.696821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.700420 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.700397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.700537 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.700479    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:20.701152 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.701109    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:20.701332 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.701313    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 21:52:20.701606 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.701588    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6lrps\""
Sep 11 21:52:20.701832 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.701807    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:20.703013 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.702825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.704620 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.704597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.704879 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.704842    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 21:52:20.705020 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.704999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r5nrg\""
Sep 11 21:52:20.705252 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.705225    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 21:52:20.705440 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.705425    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 21:52:20.707118 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.706936    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:20.707118 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.706989    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kx7x5\""
Sep 11 21:52:20.707118 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.706940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:20.707408 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.707389    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:20.707535 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.707510    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:20.709792 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.709412    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:20.709792 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.709554    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:20.712602 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.712502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.716285 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.714633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:20.716285 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.714769    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 21:52:20.716285 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.715431    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 21:52:20.716285 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.715491    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 21:52:20.716285 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.715528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tscts\""
Sep 11 21:52:20.716285 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.715848    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 21:52:20.716285 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.716035    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 21:52:20.716691 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.716523    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 21:52:20.716932 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.716910    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 21:52:20.718340 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.718310    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.719400 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.719227    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 21:52:20.720847 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.720830    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 21:52:20.720847 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.720844    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2gpf6\""
Sep 11 21:52:20.721953 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.721935    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 21:52:20.723548 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.722911    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-24tmj\""
Sep 11 21:52:20.723902 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.723054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.726009 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.725988    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 21:52:20.726203 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.726186    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 21:52:20.726266 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.726218    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5x992\""
Sep 11 21:52:20.726511 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.726496    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 21:52:20.726577 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.726532    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 21:52:20.726760 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.726723    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.728762 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.728738    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 21:52:20.728876 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.728823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.729018 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.728998    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 21:52:20.730050 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.729347    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 21:52:20.730050 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.729395    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 21:52:20.730050 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.730028    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xbbr2\""
Sep 11 21:52:20.730275 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.730259    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 21:52:20.731033 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.730991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-iptables-alerter-script\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.731119 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.731119 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731119 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-host-slash\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.731280 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8cbae59-1fe9-4805-b823-dc872f45dabc-kubelet-config\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.731280 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-run\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.731280 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-slash\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731280 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57b2\" (UniqueName: \"kubernetes.io/projected/ea986c40-eac9-451b-aec1-f6178e52c656-kube-api-access-m57b2\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.731280 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysctl-conf\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.731280 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:20.731583 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731291    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 21:52:20.731583 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aca9477-2265-43f3-b1e1-ba961826dd02-agent-certs\") pod \"konnectivity-agent-x7k89\" (UID: \"3aca9477-2265-43f3-b1e1-ba961826dd02\") " pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:20.731583 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-lib-modules\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.731583 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-run-netns\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731583 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-systemd\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731583 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731500    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4mr4m\""
Sep 11 21:52:20.731583 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4acaee8-78a0-4c04-bbab-ae5de79759a0-tmp-dir\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-cni-netd\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-env-overrides\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aca9477-2265-43f3-b1e1-ba961826dd02-konnectivity-ca\") pod \"konnectivity-agent-x7k89\" (UID: \"3aca9477-2265-43f3-b1e1-ba961826dd02\") " pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gw8k\" (UniqueName: \"kubernetes.io/projected/d2da9b9e-894f-42aa-a2cc-dde6194947ca-kube-api-access-2gw8k\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-registration-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731793    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c26zw\" (UniqueName: \"kubernetes.io/projected/f4acaee8-78a0-4c04-bbab-ae5de79759a0-kube-api-access-c26zw\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-systemd\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-ovn\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.731958 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-socket-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.731970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8cbae59-1fe9-4805-b823-dc872f45dabc-dbus\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-sys\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7mbb\" (UniqueName: \"kubernetes.io/projected/205a7047-72b1-42d0-83a8-15306e109d71-kube-api-access-g7mbb\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69ff\" (UniqueName: \"kubernetes.io/projected/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-kube-api-access-k69ff\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-etc-selinux\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/205a7047-72b1-42d0-83a8-15306e109d71-tmp\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovnkube-config\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovnkube-script-lib\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-kubernetes\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysctl-d\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-node-log\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-log-socket\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/205a7047-72b1-42d0-83a8-15306e109d71-etc-tuned\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f4acaee8-78a0-4c04-bbab-ae5de79759a0-hosts-file\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmwl\" (UniqueName: \"kubernetes.io/projected/5b7aa92b-424b-400c-bce4-30a53f9f38e7-kube-api-access-2dmwl\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:20.732465 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovn-node-metrics-cert\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-host\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-device-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-modprobe-d\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysconfig\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-etc-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-kubelet\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-systemd-units\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-cni-bin\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-sys-fs\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-var-lib-kubelet\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:20.733224 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.732829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-var-lib-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.750934 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.750909    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9ghpg"
Sep 11 21:52:20.760873 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.760331    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9ghpg"
Sep 11 21:52:20.818140 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.818106    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 21:52:20.833059 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aca9477-2265-43f3-b1e1-ba961826dd02-agent-certs\") pod \"konnectivity-agent-x7k89\" (UID: \"3aca9477-2265-43f3-b1e1-ba961826dd02\") " pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:20.833059 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-lib-modules\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.833363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-run-netns\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.833363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-lib-modules\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.833363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-systemd\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.833363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4acaee8-78a0-4c04-bbab-ae5de79759a0-tmp-dir\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.833363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-run-netns\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.833363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-cni-netd\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.833363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-systemd\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.833770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833451    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 21:52:20.833770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-env-overrides\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.833770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.833770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-cni-netd\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.833770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.833770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aca9477-2265-43f3-b1e1-ba961826dd02-konnectivity-ca\") pod \"konnectivity-agent-x7k89\" (UID: \"3aca9477-2265-43f3-b1e1-ba961826dd02\") " pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:20.833770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.833770 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4acaee8-78a0-4c04-bbab-ae5de79759a0-tmp-dir\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.834576 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.833752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.836904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hxm\" (UniqueName: \"kubernetes.io/projected/9486e46e-ec91-4007-b19d-eed2ac5784bf-kube-api-access-t5hxm\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.836973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gw8k\" (UniqueName: \"kubernetes.io/projected/d2da9b9e-894f-42aa-a2cc-dde6194947ca-kube-api-access-2gw8k\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-registration-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c26zw\" (UniqueName: \"kubernetes.io/projected/f4acaee8-78a0-4c04-bbab-ae5de79759a0-kube-api-access-c26zw\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-systemd\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-ovn\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-cnibin\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-socket-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8cbae59-1fe9-4805-b823-dc872f45dabc-dbus\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-env-overrides\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-sys\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7mbb\" (UniqueName: \"kubernetes.io/projected/205a7047-72b1-42d0-83a8-15306e109d71-kube-api-access-g7mbb\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.840480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69ff\" (UniqueName: \"kubernetes.io/projected/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-kube-api-access-k69ff\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-etc-selinux\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/205a7047-72b1-42d0-83a8-15306e109d71-tmp\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovnkube-config\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovnkube-script-lib\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-cni-multus\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-hostroot\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-kubernetes\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysctl-d\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-node-log\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-log-socket\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.837993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-system-cni-dir\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-cni-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-conf-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/205a7047-72b1-42d0-83a8-15306e109d71-etc-tuned\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f4acaee8-78a0-4c04-bbab-ae5de79759a0-hosts-file\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.841404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aca9477-2265-43f3-b1e1-ba961826dd02-konnectivity-ca\") pod \"konnectivity-agent-x7k89\" (UID: \"3aca9477-2265-43f3-b1e1-ba961826dd02\") " pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aca9477-2265-43f3-b1e1-ba961826dd02-agent-certs\") pod \"konnectivity-agent-x7k89\" (UID: \"3aca9477-2265-43f3-b1e1-ba961826dd02\") " pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-sys\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-socket-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmwl\" (UniqueName: \"kubernetes.io/projected/5b7aa92b-424b-400c-bce4-30a53f9f38e7-kube-api-access-2dmwl\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-registration-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovn-node-metrics-cert\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-k8s-cni-cncf-io\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-cni-bin\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8cbae59-1fe9-4805-b823-dc872f45dabc-dbus\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-etc-kubernetes\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-run-ovn\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-host\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.838983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-device-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-modprobe-d\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.842337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-etc-selinux\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-socket-dir-parent\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-kubelet\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-kubernetes\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-host\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-systemd\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-node-log\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysctl-d\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-daemon-config\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysconfig\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-etc-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/826cf751-e2f9-4840-b136-6ca3241d2b29-serviceca\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgns\" (UniqueName: \"kubernetes.io/projected/826cf751-e2f9-4840-b136-6ca3241d2b29-kube-api-access-8wgns\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovnkube-script-lib\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.839862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-cnibin\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-modprobe-d\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840081    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-device-dir\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f4acaee8-78a0-4c04-bbab-ae5de79759a0-hosts-file\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.843334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysconfig\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-etc-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-log-socket\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovnkube-config\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-multus-certs\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vh24\" (UniqueName: \"kubernetes.io/projected/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-kube-api-access-7vh24\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-kubelet\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-systemd-units\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-cni-bin\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-sys-fs\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-var-lib-kubelet\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-kubelet\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.840969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-var-lib-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-system-cni-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea986c40-eac9-451b-aec1-f6178e52c656-sys-fs\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-iptables-alerter-script\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.844323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-host-slash\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/826cf751-e2f9-4840-b136-6ca3241d2b29-host\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-cni-binary-copy\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8cbae59-1fe9-4805-b823-dc872f45dabc-kubelet-config\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-run\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-slash\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-netns\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57b2\" (UniqueName: \"kubernetes.io/projected/ea986c40-eac9-451b-aec1-f6178e52c656-kube-api-access-m57b2\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysctl-conf\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-iptables-alerter-script\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.841659    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-var-lib-kubelet\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.841768    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs podName:5b7aa92b-424b-400c-bce4-30a53f9f38e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:21.341738267 +0000 UTC m=+3.026684975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs") pod "network-metrics-daemon-992bd" (UID: "5b7aa92b-424b-400c-bce4-30a53f9f38e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-cni-bin\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.845295 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-var-lib-openvswitch\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.841920    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.841969    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret podName:e8cbae59-1fe9-4805-b823-dc872f45dabc nodeName:}" failed. No retries permitted until 2026-09-11 21:52:21.34195473 +0000 UTC m=+3.026901441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret") pod "global-pull-secret-syncer-cq4fq" (UID: "e8cbae59-1fe9-4805-b823-dc872f45dabc") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.841987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-run\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.842017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-slash\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.842058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8cbae59-1fe9-4805-b823-dc872f45dabc-kubelet-config\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.842112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.842164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-host-slash\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.842211    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2da9b9e-894f-42aa-a2cc-dde6194947ca-systemd-units\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.842316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/205a7047-72b1-42d0-83a8-15306e109d71-etc-sysctl-conf\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.845203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/205a7047-72b1-42d0-83a8-15306e109d71-tmp\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.845550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7mbb\" (UniqueName: \"kubernetes.io/projected/205a7047-72b1-42d0-83a8-15306e109d71-kube-api-access-g7mbb\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.846216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.845700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2da9b9e-894f-42aa-a2cc-dde6194947ca-ovn-node-metrics-cert\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.847335 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.847136    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:20.847335 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.847169    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:20.847335 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.847187    2578 projected.go:196] Error preparing data for projected volume kube-api-access-hhnw9 for pod openshift-network-diagnostics/network-check-target-9qqhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:20.847335 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.847259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/205a7047-72b1-42d0-83a8-15306e109d71-etc-tuned\") pod \"tuned-4wzr8\" (UID: \"205a7047-72b1-42d0-83a8-15306e109d71\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:20.847335 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:20.847268    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9 podName:bd8ff25a-e861-4f83-8d87-c97cd89d5075 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:21.347239464 +0000 UTC m=+3.032186180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hhnw9" (UniqueName: "kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9") pod "network-check-target-9qqhz" (UID: "bd8ff25a-e861-4f83-8d87-c97cd89d5075") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:20.848933 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.848889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69ff\" (UniqueName: \"kubernetes.io/projected/9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e-kube-api-access-k69ff\") pod \"iptables-alerter-rmt84\" (UID: \"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e\") " pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:20.849027 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.848973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gw8k\" (UniqueName: \"kubernetes.io/projected/d2da9b9e-894f-42aa-a2cc-dde6194947ca-kube-api-access-2gw8k\") pod \"ovnkube-node-pld4d\" (UID: \"d2da9b9e-894f-42aa-a2cc-dde6194947ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:20.849586 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.849543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c26zw\" (UniqueName: \"kubernetes.io/projected/f4acaee8-78a0-4c04-bbab-ae5de79759a0-kube-api-access-c26zw\") pod \"node-resolver-dj8qt\" (UID: \"f4acaee8-78a0-4c04-bbab-ae5de79759a0\") " pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:20.849666 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.849593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmwl\" (UniqueName: \"kubernetes.io/projected/5b7aa92b-424b-400c-bce4-30a53f9f38e7-kube-api-access-2dmwl\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:20.851340 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.851315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57b2\" (UniqueName: \"kubernetes.io/projected/ea986c40-eac9-451b-aec1-f6178e52c656-kube-api-access-m57b2\") pod \"aws-ebs-csi-driver-node-47p66\" (UID: \"ea986c40-eac9-451b-aec1-f6178e52c656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:20.855935 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.855886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" event={"ID":"9fec26c83d95126dedd1e7e37e73c673","Type":"ContainerStarted","Data":"dceb5c2f70124118ca63abd46bd2e353e15e4c9d8ccfbb635c44e76fb24466aa"}
Sep 11 21:52:20.857971 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.857946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"ddec20afa56710be0d0517a302df8414","Type":"ContainerStarted","Data":"206bbd4b7fb9b3b4429892cdef66a39be06d9d7c8c11c88c5f503cd6e06aec4c"}
Sep 11 21:52:20.941980 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.941941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942171 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.941988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hxm\" (UniqueName: \"kubernetes.io/projected/9486e46e-ec91-4007-b19d-eed2ac5784bf-kube-api-access-t5hxm\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942171 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-cnibin\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942171 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942171 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942171 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-cni-multus\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942171 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-hostroot\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-hostroot\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-cnibin\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-cni-multus\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-system-cni-dir\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-cni-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9486e46e-ec91-4007-b19d-eed2ac5784bf-system-cni-dir\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-conf-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-k8s-cni-cncf-io\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-cni-bin\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-cni-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-etc-kubernetes\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-socket-dir-parent\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-k8s-cni-cncf-io\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.942477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-cni-bin\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-kubelet\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-socket-dir-parent\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-var-lib-kubelet\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-conf-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-etc-kubernetes\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-daemon-config\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/826cf751-e2f9-4840-b136-6ca3241d2b29-serviceca\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgns\" (UniqueName: \"kubernetes.io/projected/826cf751-e2f9-4840-b136-6ca3241d2b29-kube-api-access-8wgns\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-cnibin\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-multus-certs\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vh24\" (UniqueName: \"kubernetes.io/projected/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-kube-api-access-7vh24\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-system-cni-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/826cf751-e2f9-4840-b136-6ca3241d2b29-host\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-cni-binary-copy\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-netns\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.942939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.943323 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-multus-certs\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-multus-daemon-config\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-system-cni-dir\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/826cf751-e2f9-4840-b136-6ca3241d2b29-serviceca\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/826cf751-e2f9-4840-b136-6ca3241d2b29-host\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-cnibin\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-host-run-netns\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9486e46e-ec91-4007-b19d-eed2ac5784bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.943982 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.943771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-cni-binary-copy\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:20.951317 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.951285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hxm\" (UniqueName: \"kubernetes.io/projected/9486e46e-ec91-4007-b19d-eed2ac5784bf-kube-api-access-t5hxm\") pod \"multus-additional-cni-plugins-5ncm8\" (UID: \"9486e46e-ec91-4007-b19d-eed2ac5784bf\") " pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:20.951471 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.951363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgns\" (UniqueName: \"kubernetes.io/projected/826cf751-e2f9-4840-b136-6ca3241d2b29-kube-api-access-8wgns\") pod \"node-ca-h8vxg\" (UID: \"826cf751-e2f9-4840-b136-6ca3241d2b29\") " pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:20.951533 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:20.951465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vh24\" (UniqueName: \"kubernetes.io/projected/f9e62c22-70fb-4a5f-9a87-4454827ecb1d-kube-api-access-7vh24\") pod \"multus-nrw8m\" (UID: \"f9e62c22-70fb-4a5f-9a87-4454827ecb1d\") " pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:21.017853 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.017771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rmt84"
Sep 11 21:52:21.040607 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.040570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66"
Sep 11 21:52:21.049946 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.049915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wzr8"
Sep 11 21:52:21.058063 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.058033    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:21.065252 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.065225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:21.073563 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.073532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dj8qt"
Sep 11 21:52:21.083534 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.083502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h8vxg"
Sep 11 21:52:21.090488 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.090462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5ncm8"
Sep 11 21:52:21.097736 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.097710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrw8m"
Sep 11 21:52:21.346113 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.346020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:21.346113 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.346082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:21.346356 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:21.346196    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:21.346356 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:21.346278    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret podName:e8cbae59-1fe9-4805-b823-dc872f45dabc nodeName:}" failed. No retries permitted until 2026-09-11 21:52:22.346257148 +0000 UTC m=+4.031203866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret") pod "global-pull-secret-syncer-cq4fq" (UID: "e8cbae59-1fe9-4805-b823-dc872f45dabc") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:21.346356 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:21.346196    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:21.346528 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:21.346399    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs podName:5b7aa92b-424b-400c-bce4-30a53f9f38e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:22.346369944 +0000 UTC m=+4.031316644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs") pod "network-metrics-daemon-992bd" (UID: "5b7aa92b-424b-400c-bce4-30a53f9f38e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:21.447028 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.446987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:21.447235 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:21.447192    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:21.447235 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:21.447222    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:21.447235 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:21.447236    2578 projected.go:196] Error preparing data for projected volume kube-api-access-hhnw9 for pod openshift-network-diagnostics/network-check-target-9qqhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:21.447397 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:21.447308    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9 podName:bd8ff25a-e861-4f83-8d87-c97cd89d5075 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:22.447287895 +0000 UTC m=+4.132234607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhnw9" (UniqueName: "kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9") pod "network-check-target-9qqhz" (UID: "bd8ff25a-e861-4f83-8d87-c97cd89d5075") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:21.764599 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.764490    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:47:21 +0000 UTC" deadline="2028-06-22 12:26:43.579363976 +0000 UTC"
Sep 11 21:52:21.764599 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:21.764540    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15590h34m21.814829715s"
Sep 11 21:52:21.934400 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:21.934370    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205a7047_72b1_42d0_83a8_15306e109d71.slice/crio-e9d490f84d1a6579685939267ee9014380fa62a94947c1a75b43105240cc69e2 WatchSource:0}: Error finding container e9d490f84d1a6579685939267ee9014380fa62a94947c1a75b43105240cc69e2: Status 404 returned error can't find the container with id e9d490f84d1a6579685939267ee9014380fa62a94947c1a75b43105240cc69e2
Sep 11 21:52:21.935297 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:21.935267    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9486e46e_ec91_4007_b19d_eed2ac5784bf.slice/crio-c512dba32ccb31756fbe2e04c98ae3517f2cf471712ff4b47aa170f59d5d9442 WatchSource:0}: Error finding container c512dba32ccb31756fbe2e04c98ae3517f2cf471712ff4b47aa170f59d5d9442: Status 404 returned error can't find the container with id c512dba32ccb31756fbe2e04c98ae3517f2cf471712ff4b47aa170f59d5d9442
Sep 11 21:52:21.936153 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:21.936023    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4acaee8_78a0_4c04_bbab_ae5de79759a0.slice/crio-c3a0eca17b2312185f1371b9d66b653c6b318218bf6aea9012cc158dcddc8eb7 WatchSource:0}: Error finding container c3a0eca17b2312185f1371b9d66b653c6b318218bf6aea9012cc158dcddc8eb7: Status 404 returned error can't find the container with id c3a0eca17b2312185f1371b9d66b653c6b318218bf6aea9012cc158dcddc8eb7
Sep 11 21:52:21.963420 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:21.963357    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826cf751_e2f9_4840_b136_6ca3241d2b29.slice/crio-821e62ee077c82e7d6290515fd2ad55021a046fc93d52b2d2049698fbd5ea8a4 WatchSource:0}: Error finding container 821e62ee077c82e7d6290515fd2ad55021a046fc93d52b2d2049698fbd5ea8a4: Status 404 returned error can't find the container with id 821e62ee077c82e7d6290515fd2ad55021a046fc93d52b2d2049698fbd5ea8a4
Sep 11 21:52:21.964526 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:21.964497    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9989c3b7_f50c_41bc_99eb_9d6ad2bfe36e.slice/crio-65820d3f2af0a04bc1b202a42cfc8f778ff7b3d98a207872df8b0da47fc9ebfa WatchSource:0}: Error finding container 65820d3f2af0a04bc1b202a42cfc8f778ff7b3d98a207872df8b0da47fc9ebfa: Status 404 returned error can't find the container with id 65820d3f2af0a04bc1b202a42cfc8f778ff7b3d98a207872df8b0da47fc9ebfa
Sep 11 21:52:21.966175 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:21.966133    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aca9477_2265_43f3_b1e1_ba961826dd02.slice/crio-721cdf356bd454a939236d8358d9e0fa8c3b1f5c80a1416a4c2e37d9277c6bb8 WatchSource:0}: Error finding container 721cdf356bd454a939236d8358d9e0fa8c3b1f5c80a1416a4c2e37d9277c6bb8: Status 404 returned error can't find the container with id 721cdf356bd454a939236d8358d9e0fa8c3b1f5c80a1416a4c2e37d9277c6bb8
Sep 11 21:52:21.967306 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:21.967281    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2da9b9e_894f_42aa_a2cc_dde6194947ca.slice/crio-b67527186ac003a367e0a45e147cfc318263636e8aa3ddd4aab42e2ebd79393c WatchSource:0}: Error finding container b67527186ac003a367e0a45e147cfc318263636e8aa3ddd4aab42e2ebd79393c: Status 404 returned error can't find the container with id b67527186ac003a367e0a45e147cfc318263636e8aa3ddd4aab42e2ebd79393c
Sep 11 21:52:22.354228 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.354188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:22.354419 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.354250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:22.354419 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.354373    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:22.354539 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.354424    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs podName:5b7aa92b-424b-400c-bce4-30a53f9f38e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:24.354408922 +0000 UTC m=+6.039355630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs") pod "network-metrics-daemon-992bd" (UID: "5b7aa92b-424b-400c-bce4-30a53f9f38e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:22.354778 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.354754    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:22.354862 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.354835    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret podName:e8cbae59-1fe9-4805-b823-dc872f45dabc nodeName:}" failed. No retries permitted until 2026-09-11 21:52:24.354818491 +0000 UTC m=+6.039765189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret") pod "global-pull-secret-syncer-cq4fq" (UID: "e8cbae59-1fe9-4805-b823-dc872f45dabc") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:22.455794 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.455558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:22.467218 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.467182    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:22.467218 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.467217    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:22.467446 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.467236    2578 projected.go:196] Error preparing data for projected volume kube-api-access-hhnw9 for pod openshift-network-diagnostics/network-check-target-9qqhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:22.467446 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.467315    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9 podName:bd8ff25a-e861-4f83-8d87-c97cd89d5075 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:24.467293757 +0000 UTC m=+6.152240466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhnw9" (UniqueName: "kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9") pod "network-check-target-9qqhz" (UID: "bd8ff25a-e861-4f83-8d87-c97cd89d5075") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:22.765570 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.765453    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:47:21 +0000 UTC" deadline="2028-02-27 13:03:01.998432509 +0000 UTC"
Sep 11 21:52:22.765570 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.765518    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12807h10m39.232919362s"
Sep 11 21:52:22.853530 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.853496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:22.853731 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.853631    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:22.854081 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.854062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:22.854197 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.854176    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:22.854265 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.854249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:22.854339 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:22.854321    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:22.890293 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.890252    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddec20afa56710be0d0517a302df8414" containerID="092a0128cd79f771cc18aa3c898b8874557f6d90f6e83bea45d17a9477415df3" exitCode=0
Sep 11 21:52:22.890491 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.890338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"ddec20afa56710be0d0517a302df8414","Type":"ContainerDied","Data":"092a0128cd79f771cc18aa3c898b8874557f6d90f6e83bea45d17a9477415df3"}
Sep 11 21:52:22.901741 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.901535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"b67527186ac003a367e0a45e147cfc318263636e8aa3ddd4aab42e2ebd79393c"}
Sep 11 21:52:22.925702 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.925580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h8vxg" event={"ID":"826cf751-e2f9-4840-b136-6ca3241d2b29","Type":"ContainerStarted","Data":"821e62ee077c82e7d6290515fd2ad55021a046fc93d52b2d2049698fbd5ea8a4"}
Sep 11 21:52:22.939077 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.938158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrw8m" event={"ID":"f9e62c22-70fb-4a5f-9a87-4454827ecb1d","Type":"ContainerStarted","Data":"7be470f7dfcb47fa8fcdc9bb5d87926b1b659de97e509ce2bcccd3d728134907"}
Sep 11 21:52:22.951299 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.951195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66" event={"ID":"ea986c40-eac9-451b-aec1-f6178e52c656","Type":"ContainerStarted","Data":"34121e0de9ea58bc884a14849ea73c72ac4a630ae70e1b56d0e289eb44f75e0d"}
Sep 11 21:52:22.967740 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.967690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wzr8" event={"ID":"205a7047-72b1-42d0-83a8-15306e109d71","Type":"ContainerStarted","Data":"e9d490f84d1a6579685939267ee9014380fa62a94947c1a75b43105240cc69e2"}
Sep 11 21:52:22.991782 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:22.991742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" event={"ID":"9fec26c83d95126dedd1e7e37e73c673","Type":"ContainerStarted","Data":"37783bff50effbb8620c74e8c9467b62e04d1143a94800cb2249e91be99a7182"}
Sep 11 21:52:23.006904 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:23.006834    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" podStartSLOduration=3.006813227 podStartE2EDuration="3.006813227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:23.00591088 +0000 UTC m=+4.690857599" watchObservedRunningTime="2026-09-11 21:52:23.006813227 +0000 UTC m=+4.691759944"
Sep 11 21:52:23.011233 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:23.011168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x7k89" event={"ID":"3aca9477-2265-43f3-b1e1-ba961826dd02","Type":"ContainerStarted","Data":"721cdf356bd454a939236d8358d9e0fa8c3b1f5c80a1416a4c2e37d9277c6bb8"}
Sep 11 21:52:23.030452 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:23.030370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rmt84" event={"ID":"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e","Type":"ContainerStarted","Data":"65820d3f2af0a04bc1b202a42cfc8f778ff7b3d98a207872df8b0da47fc9ebfa"}
Sep 11 21:52:23.049389 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:23.049351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dj8qt" event={"ID":"f4acaee8-78a0-4c04-bbab-ae5de79759a0","Type":"ContainerStarted","Data":"c3a0eca17b2312185f1371b9d66b653c6b318218bf6aea9012cc158dcddc8eb7"}
Sep 11 21:52:23.067159 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:23.067116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerStarted","Data":"c512dba32ccb31756fbe2e04c98ae3517f2cf471712ff4b47aa170f59d5d9442"}
Sep 11 21:52:24.098457 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:24.098376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"ddec20afa56710be0d0517a302df8414","Type":"ContainerStarted","Data":"985f9b172e92f0353ca4df960a68593b79e646361eae9989dfc980eaa1f58033"}
Sep 11 21:52:24.372690 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:24.372559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:24.372690 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:24.372622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:24.372925 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.372795    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:24.372925 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.372866    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs podName:5b7aa92b-424b-400c-bce4-30a53f9f38e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:28.372846481 +0000 UTC m=+10.057793192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs") pod "network-metrics-daemon-992bd" (UID: "5b7aa92b-424b-400c-bce4-30a53f9f38e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:24.373416 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.373304    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:24.373416 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.373378    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret podName:e8cbae59-1fe9-4805-b823-dc872f45dabc nodeName:}" failed. No retries permitted until 2026-09-11 21:52:28.373358378 +0000 UTC m=+10.058305087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret") pod "global-pull-secret-syncer-cq4fq" (UID: "e8cbae59-1fe9-4805-b823-dc872f45dabc") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:24.474492 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:24.473609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:24.474492 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.473938    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:24.474492 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.473990    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:24.474492 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.474005    2578 projected.go:196] Error preparing data for projected volume kube-api-access-hhnw9 for pod openshift-network-diagnostics/network-check-target-9qqhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:24.474492 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.474076    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9 podName:bd8ff25a-e861-4f83-8d87-c97cd89d5075 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:28.474054116 +0000 UTC m=+10.159000818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhnw9" (UniqueName: "kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9") pod "network-check-target-9qqhz" (UID: "bd8ff25a-e861-4f83-8d87-c97cd89d5075") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:24.851188 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:24.851083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:24.851352 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.851237    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:24.852505 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:24.851661    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:24.852505 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.851791    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:24.852505 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:24.851849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:24.852505 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:24.851941    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:26.856397 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:26.855502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:26.856397 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:26.855637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:26.856397 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:26.856064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:26.856397 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:26.856170    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:26.856397 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:26.856243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:26.856397 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:26.856340    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:28.414521 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:28.414483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:28.415039 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:28.414540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:28.415039 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.414707    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:28.415039 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.414788    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs podName:5b7aa92b-424b-400c-bce4-30a53f9f38e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:36.41476532 +0000 UTC m=+18.099712027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs") pod "network-metrics-daemon-992bd" (UID: "5b7aa92b-424b-400c-bce4-30a53f9f38e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:28.415039 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.414812    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:28.415039 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.414880    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret podName:e8cbae59-1fe9-4805-b823-dc872f45dabc nodeName:}" failed. No retries permitted until 2026-09-11 21:52:36.41486027 +0000 UTC m=+18.099806979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret") pod "global-pull-secret-syncer-cq4fq" (UID: "e8cbae59-1fe9-4805-b823-dc872f45dabc") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:28.516021 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:28.515872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:28.516276 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.516102    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:28.516276 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.516132    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:28.516276 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.516146    2578 projected.go:196] Error preparing data for projected volume kube-api-access-hhnw9 for pod openshift-network-diagnostics/network-check-target-9qqhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:28.516276 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.516197    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9 podName:bd8ff25a-e861-4f83-8d87-c97cd89d5075 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:36.516181309 +0000 UTC m=+18.201128002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhnw9" (UniqueName: "kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9") pod "network-check-target-9qqhz" (UID: "bd8ff25a-e861-4f83-8d87-c97cd89d5075") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:28.854935 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:28.854850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:28.855112 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.854977    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:28.855372 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:28.855352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:28.855474 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.855454    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:28.855534 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:28.855512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:28.855626 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:28.855596    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:30.851268 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:30.851186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:30.851780 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:30.851327    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:30.851780 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:30.851189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:30.851780 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:30.851186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:30.851780 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:30.851473    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:30.851780 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:30.851538    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:32.850521 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:32.850488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:32.850985 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:32.850528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:32.850985 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:32.850492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:32.850985 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:32.850620    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:32.850985 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:32.850692    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:32.850985 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:32.850750    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:34.850564 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:34.850521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:34.851032 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:34.850526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:34.851032 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:34.850682    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:34.851032 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:34.850527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:34.851032 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:34.850755    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:34.851032 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:34.850817    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:36.480000 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:36.479957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:36.480437 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:36.480015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:36.480437 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.480100    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:36.480437 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.480140    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:36.480437 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.480165    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret podName:e8cbae59-1fe9-4805-b823-dc872f45dabc nodeName:}" failed. No retries permitted until 2026-09-11 21:52:52.480150421 +0000 UTC m=+34.165097113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret") pod "global-pull-secret-syncer-cq4fq" (UID: "e8cbae59-1fe9-4805-b823-dc872f45dabc") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:36.480437 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.480201    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs podName:5b7aa92b-424b-400c-bce4-30a53f9f38e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:52.480183725 +0000 UTC m=+34.165130425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs") pod "network-metrics-daemon-992bd" (UID: "5b7aa92b-424b-400c-bce4-30a53f9f38e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:36.581197 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:36.581163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:36.581362 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.581311    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:36.581362 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.581330    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:36.581362 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.581339    2578 projected.go:196] Error preparing data for projected volume kube-api-access-hhnw9 for pod openshift-network-diagnostics/network-check-target-9qqhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:36.581500 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.581395    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9 podName:bd8ff25a-e861-4f83-8d87-c97cd89d5075 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:52.581377299 +0000 UTC m=+34.266324015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhnw9" (UniqueName: "kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9") pod "network-check-target-9qqhz" (UID: "bd8ff25a-e861-4f83-8d87-c97cd89d5075") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:36.851009 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:36.850925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:36.851172 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:36.850925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:36.851172 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.851060    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:36.851172 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.851083    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:36.851172 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:36.850946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:36.851373 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:36.851248    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:38.851593 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:38.851553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:38.852055 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:38.851670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:38.852055 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:38.851688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:38.852055 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:38.851681    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:38.852055 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:38.851798    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:38.852055 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:38.851878    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:40.851170 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:40.850873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:40.851909 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:40.850874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:40.851909 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:40.851259    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:40.851909 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:40.850930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:40.851909 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:40.851324    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:40.851909 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:40.851426    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:41.145175 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.145085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrw8m" event={"ID":"f9e62c22-70fb-4a5f-9a87-4454827ecb1d","Type":"ContainerStarted","Data":"285cdadb70b04347ecca8e08b930f7e4f079508211cd8c928df76ad188e98f93"}
Sep 11 21:52:41.146786 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.146732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66" event={"ID":"ea986c40-eac9-451b-aec1-f6178e52c656","Type":"ContainerStarted","Data":"a7a60218835845240b831636d59c7627cc8a50aee215d0acf273955422270e36"}
Sep 11 21:52:41.148469 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.148448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wzr8" event={"ID":"205a7047-72b1-42d0-83a8-15306e109d71","Type":"ContainerStarted","Data":"9c29a8e868f1e64392b79b5d7d8d13438cd026b3320a7afd5f79bd892a5f90df"}
Sep 11 21:52:41.149951 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.149927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x7k89" event={"ID":"3aca9477-2265-43f3-b1e1-ba961826dd02","Type":"ContainerStarted","Data":"bc3bfff87429f8719e7f1742f670aba64e2d78a2eeedf752833504b35c70749c"}
Sep 11 21:52:41.151423 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.151402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dj8qt" event={"ID":"f4acaee8-78a0-4c04-bbab-ae5de79759a0","Type":"ContainerStarted","Data":"de387c62ab52eb131f3798cc33a7f05afe1f174a3533cea2ff9efc16a26c1bce"}
Sep 11 21:52:41.152783 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.152764    2578 generic.go:356] "Generic (PLEG): container finished" podID="9486e46e-ec91-4007-b19d-eed2ac5784bf" containerID="44c55f4178c48656a0c77399eb4d0b034a64e38bc2ae8a4bce641217e1c0ba4f" exitCode=0
Sep 11 21:52:41.152864 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.152822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerDied","Data":"44c55f4178c48656a0c77399eb4d0b034a64e38bc2ae8a4bce641217e1c0ba4f"}
Sep 11 21:52:41.155838 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.155819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"9a05ecec027d89ffefdefd2611e4896e9fe50f98e384af99de2515d9bbd74529"}
Sep 11 21:52:41.155916 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.155882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"9e29ef777331cecda4aae288806cbc49059412982659b7563c3a2e27f8d47d63"}
Sep 11 21:52:41.155978 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.155919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"761eaaaef1b4ca795609f1831f80d44cb1074fdc13bbf4c511b7ffec71423ea8"}
Sep 11 21:52:41.155978 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.155961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"786ce1c04132178b65761362d6761b41bb3104d5dc2dfa96353a8c1d4e58afc3"}
Sep 11 21:52:41.156117 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.155993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"0c4b9fc8e3bf3793297746c89f68f86281710b45cc9c24cb4a1ce2056928b921"}
Sep 11 21:52:41.156117 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.156034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"2f3e9c50d33b66c414ef0ba8881797bd64a0115e458e839d5a9fefe04c623f2e"}
Sep 11 21:52:41.157177 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.157158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h8vxg" event={"ID":"826cf751-e2f9-4840-b136-6ca3241d2b29","Type":"ContainerStarted","Data":"aec1032aad94e4b22292be77e294519fa94e1fdb19ac742386f1b6d11f998dfb"}
Sep 11 21:52:41.163278 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.163227    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" podStartSLOduration=21.163208119 podStartE2EDuration="21.163208119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:24.114485879 +0000 UTC m=+5.799432598" watchObservedRunningTime="2026-09-11 21:52:41.163208119 +0000 UTC m=+22.848154836"
Sep 11 21:52:41.163472 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.163438    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nrw8m" podStartSLOduration=2.717238535 podStartE2EDuration="21.163428672s" totalImagesPullingTime="18.446190137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:41.162653609 +0000 UTC m=+22.847600322" watchObservedRunningTime="2026-09-11 21:52:41.163428672 +0000 UTC m=+22.848375387"
Sep 11 21:52:41.178399 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.178352    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4wzr8" podStartSLOduration=2.737921037 podStartE2EDuration="21.178338478s" totalImagesPullingTime="18.440417441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:41.178244692 +0000 UTC m=+22.863191405" watchObservedRunningTime="2026-09-11 21:52:41.178338478 +0000 UTC m=+22.863285192"
Sep 11 21:52:41.191496 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.191451    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x7k89" podStartSLOduration=3.034448048 podStartE2EDuration="21.191434121s" totalImagesPullingTime="18.156986073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:41.19091002 +0000 UTC m=+22.875856754" watchObservedRunningTime="2026-09-11 21:52:41.191434121 +0000 UTC m=+22.876380836"
Sep 11 21:52:41.206008 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.205939    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dj8qt" podStartSLOduration=2.796014246 podStartE2EDuration="21.205918563s" totalImagesPullingTime="18.409904317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:41.205261478 +0000 UTC m=+22.890208192" watchObservedRunningTime="2026-09-11 21:52:41.205918563 +0000 UTC m=+22.890865277"
Sep 11 21:52:41.241806 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.241744    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h8vxg" podStartSLOduration=2.860793806 podStartE2EDuration="21.241723285s" totalImagesPullingTime="18.380929479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:41.220869142 +0000 UTC m=+22.905815858" watchObservedRunningTime="2026-09-11 21:52:41.241723285 +0000 UTC m=+22.926670001"
Sep 11 21:52:41.736713 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.736684    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 21:52:41.927659 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.927497    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T21:52:41.736708986Z","UUID":"75efa350-e2cd-4d2e-8366-8ae8c0f6f33e","Handler":null,"Name":"","Endpoint":""}
Sep 11 21:52:41.929173 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.929154    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 21:52:41.929299 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:41.929186    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 21:52:42.161195 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:42.161152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rmt84" event={"ID":"9989c3b7-f50c-41bc-99eb-9d6ad2bfe36e","Type":"ContainerStarted","Data":"c5b0c67db197c030551115e6e6d0607ed0732cd0511a3aa4b5c4351a3cbb6eee"}
Sep 11 21:52:42.164575 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:42.164172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66" event={"ID":"ea986c40-eac9-451b-aec1-f6178e52c656","Type":"ContainerStarted","Data":"9b6c8490002f5a9bb4fbec7ab93404bb92db820119d829939d2bb573a1f029b9"}
Sep 11 21:52:42.175155 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:42.175106    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rmt84" podStartSLOduration=3.7882832889999998 podStartE2EDuration="22.175085418s" totalImagesPullingTime="18.386802129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:42.174985601 +0000 UTC m=+23.859932315" watchObservedRunningTime="2026-09-11 21:52:42.175085418 +0000 UTC m=+23.860032133"
Sep 11 21:52:42.712802 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:42.712768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:42.713437 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:42.713417    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:42.850525 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:42.850496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:42.850691 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:42.850496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:42.850691 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:42.850646    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:42.850691 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:42.850659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:42.850856 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:42.850787    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:42.850927 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:42.850897    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:43.176034 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:43.175992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"3d99ad1e45374ca20d66f7e1dc9678d6e2a524732ad893edbe0a26bb3328c63b"}
Sep 11 21:52:43.178817 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:43.178786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66" event={"ID":"ea986c40-eac9-451b-aec1-f6178e52c656","Type":"ContainerStarted","Data":"75ccdb7c3ee3c454400fb2e30020ca181264ca8b125aeea451f9a37e0f2d32a6"}
Sep 11 21:52:43.179459 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:43.179438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:43.179954 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:43.179914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x7k89"
Sep 11 21:52:43.195096 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:43.195045    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47p66" podStartSLOduration=2.755030083 podStartE2EDuration="23.195026007s" totalImagesPullingTime="20.439995924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:43.194490055 +0000 UTC m=+24.879436782" watchObservedRunningTime="2026-09-11 21:52:43.195026007 +0000 UTC m=+24.879972723"
Sep 11 21:52:44.851410 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:44.851374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:44.851887 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:44.851374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:44.851887 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:44.851516    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:44.851887 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:44.851374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:44.851887 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:44.851592    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:44.851887 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:44.851694    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:46.186669 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.186450    2578 generic.go:356] "Generic (PLEG): container finished" podID="9486e46e-ec91-4007-b19d-eed2ac5784bf" containerID="a69064ea805736d1ee56a4773f57986b7b299b886f240d7fea4c4d7895c9a0fa" exitCode=0
Sep 11 21:52:46.187480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.186538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerDied","Data":"a69064ea805736d1ee56a4773f57986b7b299b886f240d7fea4c4d7895c9a0fa"}
Sep 11 21:52:46.190771 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.190749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" event={"ID":"d2da9b9e-894f-42aa-a2cc-dde6194947ca","Type":"ContainerStarted","Data":"425e322d82b816a81390586e6270b5c5dd7b3dae50632ce2b80627e1cb71b04d"}
Sep 11 21:52:46.190988 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.190965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:46.209220 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.209187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:46.228029 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.227984    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d" podStartSLOduration=7.78085831 podStartE2EDuration="26.227971005s" totalImagesPullingTime="18.447112695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:46.227273231 +0000 UTC m=+27.912219936" watchObservedRunningTime="2026-09-11 21:52:46.227971005 +0000 UTC m=+27.912917718"
Sep 11 21:52:46.850960 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.850719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:46.851179 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.850719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:46.851179 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:46.851067    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:46.851301 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:46.850719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:46.851301 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:46.851185    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:46.851407 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:46.851308    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:47.157542 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.157442    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cq4fq"]
Sep 11 21:52:47.160832 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.160805    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9qqhz"]
Sep 11 21:52:47.161487 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.161458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-992bd"]
Sep 11 21:52:47.193212 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.193072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:47.193212 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.193078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:47.193212 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:47.193207    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:47.193759 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.193078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:47.193759 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:47.193308    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:47.193759 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:47.193454    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:47.193907 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.193821    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:47.193907 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.193901    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:47.209339 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:47.209320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:52:48.197353 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:48.197319    2578 generic.go:356] "Generic (PLEG): container finished" podID="9486e46e-ec91-4007-b19d-eed2ac5784bf" containerID="d42f0eaef9c7e1d2cac73b6bf367571e06a8b3062b6be75ff8fc9696a9038832" exitCode=0
Sep 11 21:52:48.197900 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:48.197411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerDied","Data":"d42f0eaef9c7e1d2cac73b6bf367571e06a8b3062b6be75ff8fc9696a9038832"}
Sep 11 21:52:48.851935 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:48.851904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:48.852102 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:48.851990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:48.852102 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:48.851992    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:48.852179 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:48.852114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:48.852217 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:48.852112    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:48.852217 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:48.852201    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:50.204527 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:50.204343    2578 generic.go:356] "Generic (PLEG): container finished" podID="9486e46e-ec91-4007-b19d-eed2ac5784bf" containerID="c693c402ee8afafbee32f20ad181ea46f11f225db811d7298590f4895988cf8c" exitCode=0
Sep 11 21:52:50.204527 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:50.204429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerDied","Data":"c693c402ee8afafbee32f20ad181ea46f11f225db811d7298590f4895988cf8c"}
Sep 11 21:52:50.850968 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:50.850906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:50.850968 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:50.850931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:50.851221 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:50.850906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:50.851221 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:50.851058    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:50.851221 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:50.851183    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:50.851366 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:50.851261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:52.507868 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:52.507832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:52.507868 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:52.507874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:52.508443 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.507999    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:52.508443 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.508001    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:52.508443 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.508062    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs podName:5b7aa92b-424b-400c-bce4-30a53f9f38e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:24.50804627 +0000 UTC m=+66.192992962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs") pod "network-metrics-daemon-992bd" (UID: "5b7aa92b-424b-400c-bce4-30a53f9f38e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:52.508443 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.508077    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret podName:e8cbae59-1fe9-4805-b823-dc872f45dabc nodeName:}" failed. No retries permitted until 2026-09-11 21:53:24.508069571 +0000 UTC m=+66.193016263 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret") pod "global-pull-secret-syncer-cq4fq" (UID: "e8cbae59-1fe9-4805-b823-dc872f45dabc") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:52.609211 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:52.609173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:52.609393 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.609336    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:52.609393 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.609357    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:52.609393 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.609368    2578 projected.go:196] Error preparing data for projected volume kube-api-access-hhnw9 for pod openshift-network-diagnostics/network-check-target-9qqhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:52.609560 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.609433    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9 podName:bd8ff25a-e861-4f83-8d87-c97cd89d5075 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:24.609412129 +0000 UTC m=+66.294358842 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhnw9" (UniqueName: "kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9") pod "network-check-target-9qqhz" (UID: "bd8ff25a-e861-4f83-8d87-c97cd89d5075") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:52.851158 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:52.851063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:52.851325 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:52.851063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:52.851325 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.851209    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-992bd" podUID="5b7aa92b-424b-400c-bce4-30a53f9f38e7"
Sep 11 21:52:52.851325 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:52.851077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:52.851325 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.851287    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cq4fq" podUID="e8cbae59-1fe9-4805-b823-dc872f45dabc"
Sep 11 21:52:52.851325 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:52.851318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qqhz" podUID="bd8ff25a-e861-4f83-8d87-c97cd89d5075"
Sep 11 21:52:53.124830 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.124743    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeReady"
Sep 11 21:52:53.124993 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.124896    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 21:52:53.159470 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.159433    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7455f658d9-5pblw"]
Sep 11 21:52:53.164755 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.164730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.167663 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.167235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 21:52:53.167663 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.167468    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5hpf\""
Sep 11 21:52:53.167663 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.167503    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 21:52:53.169043 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.168071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 21:52:53.171327 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.171304    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mjmp7"]
Sep 11 21:52:53.174733 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.174707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 21:52:53.177750 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.177728    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9tnsj"]
Sep 11 21:52:53.177870 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.177827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.179929 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.179896    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 21:52:53.180090 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.180073    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 21:52:53.180265 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.180139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8zqrn\""
Sep 11 21:52:53.181525 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.181504    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7455f658d9-5pblw"]
Sep 11 21:52:53.181663 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.181602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tnsj"
Sep 11 21:52:53.184150 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.184116    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mjmp7"]
Sep 11 21:52:53.184309 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.184290    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 21:52:53.184667 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.184650    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 21:52:53.184904 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.184884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8rpn9\""
Sep 11 21:52:53.184971 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.184935    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 21:52:53.191272 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.191252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tnsj"]
Sep 11 21:52:53.314148 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9747109c-ab47-4d2d-8297-9b8b55d19d35-config-volume\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.314148 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.314388 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-certificates\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.314388 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-installation-pull-secrets\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.314388 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9747109c-ab47-4d2d-8297-9b8b55d19d35-tmp-dir\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.314388 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbr56\" (UniqueName: \"kubernetes.io/projected/6e0f6fc4-3214-42ba-9777-62e327c1ded2-kube-api-access-rbr56\") pod \"ingress-canary-9tnsj\" (UID: \"6e0f6fc4-3214-42ba-9777-62e327c1ded2\") " pod="openshift-ingress-canary/ingress-canary-9tnsj"
Sep 11 21:52:53.314548 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnd4\" (UniqueName: \"kubernetes.io/projected/9747109c-ab47-4d2d-8297-9b8b55d19d35-kube-api-access-jxnd4\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.314548 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-image-registry-private-configuration\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.314548 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.314548 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e0f6fc4-3214-42ba-9777-62e327c1ded2-cert\") pod \"ingress-canary-9tnsj\" (UID: \"6e0f6fc4-3214-42ba-9777-62e327c1ded2\") " pod="openshift-ingress-canary/ingress-canary-9tnsj"
Sep 11 21:52:53.314548 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-trusted-ca\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.314548 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0145b0f-57d6-475c-8517-77fb08cd5428-ca-trust-extracted\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.314799 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vssph\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-kube-api-access-vssph\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.314799 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.314585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-bound-sa-token\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.415448 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-certificates\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.415618 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-installation-pull-secrets\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.415618 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9747109c-ab47-4d2d-8297-9b8b55d19d35-tmp-dir\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.415618 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr56\" (UniqueName: \"kubernetes.io/projected/6e0f6fc4-3214-42ba-9777-62e327c1ded2-kube-api-access-rbr56\") pod \"ingress-canary-9tnsj\" (UID: \"6e0f6fc4-3214-42ba-9777-62e327c1ded2\") " pod="openshift-ingress-canary/ingress-canary-9tnsj"
Sep 11 21:52:53.415618 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnd4\" (UniqueName: \"kubernetes.io/projected/9747109c-ab47-4d2d-8297-9b8b55d19d35-kube-api-access-jxnd4\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.415860 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-image-registry-private-configuration\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.415860 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.415860 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e0f6fc4-3214-42ba-9777-62e327c1ded2-cert\") pod \"ingress-canary-9tnsj\" (UID: \"6e0f6fc4-3214-42ba-9777-62e327c1ded2\") " pod="openshift-ingress-canary/ingress-canary-9tnsj"
Sep 11 21:52:53.415989 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.415871    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:53.415989 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.415893    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7455f658d9-5pblw: secret "image-registry-tls" not found
Sep 11 21:52:53.415989 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-trusted-ca\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.415989 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.415952    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls podName:f0145b0f-57d6-475c-8517-77fb08cd5428 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:53.915933982 +0000 UTC m=+35.600880684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls") pod "image-registry-7455f658d9-5pblw" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428") : secret "image-registry-tls" not found
Sep 11 21:52:53.415989 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9747109c-ab47-4d2d-8297-9b8b55d19d35-tmp-dir\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.415989 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.415984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0145b0f-57d6-475c-8517-77fb08cd5428-ca-trust-extracted\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.416254 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.416013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vssph\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-kube-api-access-vssph\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.416254 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.416054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-bound-sa-token\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.416254 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.416084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9747109c-ab47-4d2d-8297-9b8b55d19d35-config-volume\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.416254 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.416086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-certificates\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.416254 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.416136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.416254 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.416251    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:53.416532 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.416306    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls podName:9747109c-ab47-4d2d-8297-9b8b55d19d35 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:53.916290809 +0000 UTC m=+35.601237515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls") pod "dns-default-mjmp7" (UID: "9747109c-ab47-4d2d-8297-9b8b55d19d35") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:53.416532 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.416354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0145b0f-57d6-475c-8517-77fb08cd5428-ca-trust-extracted\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.416790 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.416771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9747109c-ab47-4d2d-8297-9b8b55d19d35-config-volume\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.417149 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.417128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-trusted-ca\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.420337 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.420314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-installation-pull-secrets\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.420444 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.420387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e0f6fc4-3214-42ba-9777-62e327c1ded2-cert\") pod \"ingress-canary-9tnsj\" (UID: \"6e0f6fc4-3214-42ba-9777-62e327c1ded2\") " pod="openshift-ingress-canary/ingress-canary-9tnsj"
Sep 11 21:52:53.420444 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.420361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-image-registry-private-configuration\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.424280 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.424255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-bound-sa-token\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.424966 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.424931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnd4\" (UniqueName: \"kubernetes.io/projected/9747109c-ab47-4d2d-8297-9b8b55d19d35-kube-api-access-jxnd4\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.425336 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.425317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr56\" (UniqueName: \"kubernetes.io/projected/6e0f6fc4-3214-42ba-9777-62e327c1ded2-kube-api-access-rbr56\") pod \"ingress-canary-9tnsj\" (UID: \"6e0f6fc4-3214-42ba-9777-62e327c1ded2\") " pod="openshift-ingress-canary/ingress-canary-9tnsj"
Sep 11 21:52:53.425513 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.425497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vssph\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-kube-api-access-vssph\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.495740 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.495706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tnsj"
Sep 11 21:52:53.921406 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.921373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:53.926852 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:53.921425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:53.926852 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.921558    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:53.926852 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.921588    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:53.926852 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.921602    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7455f658d9-5pblw: secret "image-registry-tls" not found
Sep 11 21:52:53.926852 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.921697    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls podName:f0145b0f-57d6-475c-8517-77fb08cd5428 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:54.921673074 +0000 UTC m=+36.606619781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls") pod "image-registry-7455f658d9-5pblw" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428") : secret "image-registry-tls" not found
Sep 11 21:52:53.926852 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:53.921716    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls podName:9747109c-ab47-4d2d-8297-9b8b55d19d35 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:54.921707917 +0000 UTC m=+36.606654609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls") pod "dns-default-mjmp7" (UID: "9747109c-ab47-4d2d-8297-9b8b55d19d35") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:54.218461 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.218295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerStarted","Data":"8eb84b14ef43af8b35787a72ca49b6ccb7a7264c4f34d0ce96d0d110d80c26f2"}
Sep 11 21:52:54.328309 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.328270    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tnsj"]
Sep 11 21:52:54.359594 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:54.359555    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0f6fc4_3214_42ba_9777_62e327c1ded2.slice/crio-275a78ae1872f34ff82a7a7f5241b23efbce53276837393f29632e8995c3778e WatchSource:0}: Error finding container 275a78ae1872f34ff82a7a7f5241b23efbce53276837393f29632e8995c3778e: Status 404 returned error can't find the container with id 275a78ae1872f34ff82a7a7f5241b23efbce53276837393f29632e8995c3778e
Sep 11 21:52:54.850709 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.850609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:52:54.850861 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.850609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:52:54.850861 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.850609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:52:54.853624 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.853600    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:52:54.853771 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.853627    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:52:54.854101 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.854079    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:52:54.854234 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.854116    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bfmc\""
Sep 11 21:52:54.854234 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.854174    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qsk4f\""
Sep 11 21:52:54.854490 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.854325    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:52:54.928706 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.928670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:54.929285 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:54.928766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:54.929285 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:54.928834    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:54.929285 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:54.928856    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7455f658d9-5pblw: secret "image-registry-tls" not found
Sep 11 21:52:54.929285 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:54.928878    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:54.929285 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:54.928945    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls podName:f0145b0f-57d6-475c-8517-77fb08cd5428 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:56.928922565 +0000 UTC m=+38.613869273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls") pod "image-registry-7455f658d9-5pblw" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428") : secret "image-registry-tls" not found
Sep 11 21:52:54.929285 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:54.928962    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls podName:9747109c-ab47-4d2d-8297-9b8b55d19d35 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:56.928953559 +0000 UTC m=+38.613900253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls") pod "dns-default-mjmp7" (UID: "9747109c-ab47-4d2d-8297-9b8b55d19d35") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:55.224648 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:55.224569    2578 generic.go:356] "Generic (PLEG): container finished" podID="9486e46e-ec91-4007-b19d-eed2ac5784bf" containerID="8eb84b14ef43af8b35787a72ca49b6ccb7a7264c4f34d0ce96d0d110d80c26f2" exitCode=0
Sep 11 21:52:55.224856 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:55.224662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerDied","Data":"8eb84b14ef43af8b35787a72ca49b6ccb7a7264c4f34d0ce96d0d110d80c26f2"}
Sep 11 21:52:55.226064 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:55.226018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tnsj" event={"ID":"6e0f6fc4-3214-42ba-9777-62e327c1ded2","Type":"ContainerStarted","Data":"275a78ae1872f34ff82a7a7f5241b23efbce53276837393f29632e8995c3778e"}
Sep 11 21:52:56.232873 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:56.232832    2578 generic.go:356] "Generic (PLEG): container finished" podID="9486e46e-ec91-4007-b19d-eed2ac5784bf" containerID="eebda3d47f32c259e146644103ca637d4b875dee68df34a684bc248457c1f330" exitCode=0
Sep 11 21:52:56.233430 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:56.232887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerDied","Data":"eebda3d47f32c259e146644103ca637d4b875dee68df34a684bc248457c1f330"}
Sep 11 21:52:56.946280 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:56.946086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:52:56.946470 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:56.946229    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:52:56.946470 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:56.946343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:52:56.946470 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:56.946350    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7455f658d9-5pblw: secret "image-registry-tls" not found
Sep 11 21:52:56.946470 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:56.946401    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:52:56.946470 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:56.946456    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls podName:f0145b0f-57d6-475c-8517-77fb08cd5428 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:00.946437518 +0000 UTC m=+42.631384224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls") pod "image-registry-7455f658d9-5pblw" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428") : secret "image-registry-tls" not found
Sep 11 21:52:56.946710 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:56.946476    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls podName:9747109c-ab47-4d2d-8297-9b8b55d19d35 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:00.946467042 +0000 UTC m=+42.631413947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls") pod "dns-default-mjmp7" (UID: "9747109c-ab47-4d2d-8297-9b8b55d19d35") : secret "dns-default-metrics-tls" not found
Sep 11 21:52:57.239171 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:57.239135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" event={"ID":"9486e46e-ec91-4007-b19d-eed2ac5784bf","Type":"ContainerStarted","Data":"f9bbdbfa664e14c2f276f5eb1647ccd5a166bdef3b2ebeda6e26ef4ee9c66e55"}
Sep 11 21:52:57.242589 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:57.242559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tnsj" event={"ID":"6e0f6fc4-3214-42ba-9777-62e327c1ded2","Type":"ContainerStarted","Data":"bc4d8e3a00fd8c4e8c3d237159b9b4c20862489e34fbdb0692d49be39dfb2d32"}
Sep 11 21:52:57.260363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:57.260320    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5ncm8" podStartSLOduration=8.590932592 podStartE2EDuration="37.26030646s" totalImagesPullingTime="28.669373868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:57.259058347 +0000 UTC m=+38.944005061" watchObservedRunningTime="2026-09-11 21:52:57.26030646 +0000 UTC m=+38.945253175"
Sep 11 21:52:57.274692 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:57.274626    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9tnsj" podStartSLOduration=2.295486869 podStartE2EDuration="4.274607454s" totalImagesPullingTime="1.979120585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:57.274170055 +0000 UTC m=+38.959116769" watchObservedRunningTime="2026-09-11 21:52:57.274607454 +0000 UTC m=+38.959554168"
Sep 11 21:52:58.392366 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.392330    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6l6gw"]
Sep 11 21:52:58.434389 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.434347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6l6gw"]
Sep 11 21:52:58.434544 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.434474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.436904 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.436879    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k6pq6\""
Sep 11 21:52:58.436904 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.436881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 21:52:58.437114 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.436881    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 21:52:58.437331 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.437315    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 21:52:58.437383 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.437317    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 21:52:58.457453 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.457400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-crio-socket\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.457630 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.457491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-data-volume\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.457630 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.457583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.457753 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.457676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.457753 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.457729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzf72\" (UniqueName: \"kubernetes.io/projected/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-api-access-jzf72\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.558909 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.558867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.559104 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.558925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.559104 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.558959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzf72\" (UniqueName: \"kubernetes.io/projected/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-api-access-jzf72\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.559104 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.559003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-crio-socket\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.559104 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.559027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-data-volume\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.559104 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:58.559065    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:52:58.559104 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:58.559093    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:52:58.559343 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:58.559178    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:59.059154119 +0000 UTC m=+40.744100816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:52:58.559343 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:58.559211    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:59.059200204 +0000 UTC m=+40.744146898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : configmap "kube-rbac-proxy" not found
Sep 11 21:52:58.559343 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.559266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-crio-socket\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.559343 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.559322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-data-volume\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:58.568254 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:58.568231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzf72\" (UniqueName: \"kubernetes.io/projected/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-api-access-jzf72\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:59.030723 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.030690    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758"]
Sep 11 21:52:59.055139 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.055100    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758"]
Sep 11 21:52:59.055276 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.055230    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758"
Sep 11 21:52:59.057500 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.057481    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 21:52:59.058007 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.057990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:59.058056 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.057990    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sqxv7\""
Sep 11 21:52:59.062462 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.062433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:59.062563 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.062511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pc4\" (UniqueName: \"kubernetes.io/projected/00997be9-004e-4c41-b42c-82bb1783d84c-kube-api-access-g5pc4\") pod \"migrator-8b4d697c5-lz758\" (UID: \"00997be9-004e-4c41-b42c-82bb1783d84c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758"
Sep 11 21:52:59.062563 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.062556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:52:59.062692 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:59.062553    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:52:59.062692 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:59.062671    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:00.062624844 +0000 UTC m=+41.747571548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : configmap "kube-rbac-proxy" not found
Sep 11 21:52:59.062692 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:59.062688    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:52:59.062805 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:52:59.062737    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:00.062723623 +0000 UTC m=+41.747670325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:52:59.163081 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.163041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pc4\" (UniqueName: \"kubernetes.io/projected/00997be9-004e-4c41-b42c-82bb1783d84c-kube-api-access-g5pc4\") pod \"migrator-8b4d697c5-lz758\" (UID: \"00997be9-004e-4c41-b42c-82bb1783d84c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758"
Sep 11 21:52:59.170186 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.170160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pc4\" (UniqueName: \"kubernetes.io/projected/00997be9-004e-4c41-b42c-82bb1783d84c-kube-api-access-g5pc4\") pod \"migrator-8b4d697c5-lz758\" (UID: \"00997be9-004e-4c41-b42c-82bb1783d84c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758"
Sep 11 21:52:59.364304 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.364219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758"
Sep 11 21:52:59.645919 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:52:59.645842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758"]
Sep 11 21:52:59.649155 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:52:59.649131    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00997be9_004e_4c41_b42c_82bb1783d84c.slice/crio-e0a7ec5217a76ffd14257e63722d1dabb85fb15b0122811b81a883ce1e293202 WatchSource:0}: Error finding container e0a7ec5217a76ffd14257e63722d1dabb85fb15b0122811b81a883ce1e293202: Status 404 returned error can't find the container with id e0a7ec5217a76ffd14257e63722d1dabb85fb15b0122811b81a883ce1e293202
Sep 11 21:53:00.052157 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:00.052134    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dj8qt_f4acaee8-78a0-4c04-bbab-ae5de79759a0/dns-node-resolver/0.log"
Sep 11 21:53:00.069976 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:00.069930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:00.070139 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:00.070048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:00.070236 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.070219    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:53:00.070338 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.070320    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:02.070294718 +0000 UTC m=+43.755241428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : configmap "kube-rbac-proxy" not found
Sep 11 21:53:00.070854 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.070834    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:00.070948 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.070908    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:02.070883794 +0000 UTC m=+43.755830490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:00.251673 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:00.251615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758" event={"ID":"00997be9-004e-4c41-b42c-82bb1783d84c","Type":"ContainerStarted","Data":"e0a7ec5217a76ffd14257e63722d1dabb85fb15b0122811b81a883ce1e293202"}
Sep 11 21:53:00.976411 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:00.976373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:53:00.976905 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:00.976434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:53:00.976905 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.976551    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:53:00.976905 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.976550    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:53:00.976905 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.976562    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7455f658d9-5pblw: secret "image-registry-tls" not found
Sep 11 21:53:00.976905 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.976673    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls podName:9747109c-ab47-4d2d-8297-9b8b55d19d35 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:08.97662605 +0000 UTC m=+50.661572747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls") pod "dns-default-mjmp7" (UID: "9747109c-ab47-4d2d-8297-9b8b55d19d35") : secret "dns-default-metrics-tls" not found
Sep 11 21:53:00.976905 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:00.976698    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls podName:f0145b0f-57d6-475c-8517-77fb08cd5428 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:08.976686366 +0000 UTC m=+50.661633058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls") pod "image-registry-7455f658d9-5pblw" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428") : secret "image-registry-tls" not found
Sep 11 21:53:01.063556 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:01.063528    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h8vxg_826cf751-e2f9-4840-b136-6ca3241d2b29/node-ca/0.log"
Sep 11 21:53:01.255469 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:01.255433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758" event={"ID":"00997be9-004e-4c41-b42c-82bb1783d84c","Type":"ContainerStarted","Data":"7af6560bc039f1e21500ac3175f19d1c10fc0909c83ef7501ce2827c380bdc03"}
Sep 11 21:53:01.652549 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:01.652458    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tnsj_6e0f6fc4-3214-42ba-9777-62e327c1ded2/serve-healthcheck-canary/0.log"
Sep 11 21:53:02.085389 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:02.085355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:02.085814 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:02.085405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:02.085814 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:02.085484    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:53:02.085814 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:02.085518    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:02.085814 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:02.085536    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:06.085523086 +0000 UTC m=+47.770469778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : configmap "kube-rbac-proxy" not found
Sep 11 21:53:02.085814 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:02.085573    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:06.085559547 +0000 UTC m=+47.770506238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:02.259545 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:02.259511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758" event={"ID":"00997be9-004e-4c41-b42c-82bb1783d84c","Type":"ContainerStarted","Data":"d05c8b4e07988c60882d9dbb68e9d47e5c96dd7d5622fb14d2c561a709b10e45"}
Sep 11 21:53:02.275070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:02.275018    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-lz758" podStartSLOduration=1.791955798 podStartE2EDuration="3.275001949s" totalImagesPullingTime="1.483046151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:02.274554876 +0000 UTC m=+43.959501601" watchObservedRunningTime="2026-09-11 21:53:02.275001949 +0000 UTC m=+43.959948654"
Sep 11 21:53:06.120065 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:06.120018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:06.120594 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:06.120139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:06.120594 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:06.120274    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:06.120594 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:06.120361    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls podName:6c0d5d5a-e598-444e-b63a-208bafb0aaa5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:14.120338545 +0000 UTC m=+55.805285258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6l6gw" (UID: "6c0d5d5a-e598-444e-b63a-208bafb0aaa5") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:06.120806 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:06.120606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:09.042429 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:09.042381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:53:09.042862 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:09.042443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls\") pod \"image-registry-7455f658d9-5pblw\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") " pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:53:09.042862 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:09.042542    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:53:09.042862 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:09.042570    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:53:09.042862 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:09.042580    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7455f658d9-5pblw: secret "image-registry-tls" not found
Sep 11 21:53:09.042862 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:09.042618    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls podName:9747109c-ab47-4d2d-8297-9b8b55d19d35 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:25.042599737 +0000 UTC m=+66.727546429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls") pod "dns-default-mjmp7" (UID: "9747109c-ab47-4d2d-8297-9b8b55d19d35") : secret "dns-default-metrics-tls" not found
Sep 11 21:53:09.042862 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:09.042649    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls podName:f0145b0f-57d6-475c-8517-77fb08cd5428 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:25.042627171 +0000 UTC m=+66.727573863 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls") pod "image-registry-7455f658d9-5pblw" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428") : secret "image-registry-tls" not found
Sep 11 21:53:14.186464 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:14.186425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:14.188756 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:14.188734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c0d5d5a-e598-444e-b63a-208bafb0aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6l6gw\" (UID: \"6c0d5d5a-e598-444e-b63a-208bafb0aaa5\") " pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:14.343282 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:14.343244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6l6gw"
Sep 11 21:53:14.612080 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:14.611995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6l6gw"]
Sep 11 21:53:14.615477 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:14.615442    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0d5d5a_e598_444e_b63a_208bafb0aaa5.slice/crio-f1554f31daa54e195511bfbfcd65223706d83212050da960f3ddb19f3f9eba79 WatchSource:0}: Error finding container f1554f31daa54e195511bfbfcd65223706d83212050da960f3ddb19f3f9eba79: Status 404 returned error can't find the container with id f1554f31daa54e195511bfbfcd65223706d83212050da960f3ddb19f3f9eba79
Sep 11 21:53:15.289884 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:15.289846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6l6gw" event={"ID":"6c0d5d5a-e598-444e-b63a-208bafb0aaa5","Type":"ContainerStarted","Data":"6a2317f23de89998b356176eac00e4d4cb4846d0914edd97047388fb6725b49d"}
Sep 11 21:53:15.290306 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:15.289908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6l6gw" event={"ID":"6c0d5d5a-e598-444e-b63a-208bafb0aaa5","Type":"ContainerStarted","Data":"f1554f31daa54e195511bfbfcd65223706d83212050da960f3ddb19f3f9eba79"}
Sep 11 21:53:16.294206 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:16.294174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6l6gw" event={"ID":"6c0d5d5a-e598-444e-b63a-208bafb0aaa5","Type":"ContainerStarted","Data":"9b13183eea7484587f3eabc534271d43c8ac3a9f5872c27a065e72a52f2776ee"}
Sep 11 21:53:17.298777 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:17.298729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6l6gw" event={"ID":"6c0d5d5a-e598-444e-b63a-208bafb0aaa5","Type":"ContainerStarted","Data":"79415d2069fb1fc1305d41014477742c0a9bd2e2c4e122a27080f67e88f24eaa"}
Sep 11 21:53:17.316270 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:17.316221    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6l6gw" podStartSLOduration=17.359688272 podStartE2EDuration="19.316205338s" totalImagesPullingTime="1.956517066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:17.315534088 +0000 UTC m=+59.000480801" watchObservedRunningTime="2026-09-11 21:53:17.316205338 +0000 UTC m=+59.001152051"
Sep 11 21:53:19.220059 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:19.220029    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pld4d"
Sep 11 21:53:21.289011 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.288974    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"]
Sep 11 21:53:21.291624 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.291594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.293940 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.293912    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 21:53:21.294132 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.294108    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 21:53:21.294283 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.294164    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 21:53:21.294366 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.294306    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 21:53:21.294554 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.294537    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 21:53:21.294554 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.294544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 21:53:21.294820 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.294807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 21:53:21.318835 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.318803    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"]
Sep 11 21:53:21.340947 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.340913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.341140 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.340955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.341140 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.341037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-ca\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.341140 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.341076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.341140 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.341103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-hub\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.341140 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.341119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6g97\" (UniqueName: \"kubernetes.io/projected/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-kube-api-access-v6g97\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.374209 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.374176    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7455f658d9-5pblw"]
Sep 11 21:53:21.374399 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:21.374379    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7455f658d9-5pblw" podUID="f0145b0f-57d6-475c-8517-77fb08cd5428"
Sep 11 21:53:21.441906 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.441872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.441906 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.441909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.442145 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.441935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-ca\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.442145 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.441951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.442145 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.441969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-hub\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.442145 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.441994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6g97\" (UniqueName: \"kubernetes.io/projected/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-kube-api-access-v6g97\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.442804 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.442783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.444421 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.444389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-ca\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.444561 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.444542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.444795 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.444776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-hub\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.444901 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.444884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.452837 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.452813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6g97\" (UniqueName: \"kubernetes.io/projected/a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5-kube-api-access-v6g97\") pod \"cluster-proxy-proxy-agent-76dfd49fc6-dmfxg\" (UID: \"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.609389 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.609305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"
Sep 11 21:53:21.877093 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:21.877023    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg"]
Sep 11 21:53:21.880109 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:21.880084    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0f7a5c3_9b6a_430d_a5e7_9e75421a3dc5.slice/crio-e92a5219756fe1139e6a812bfaeb2c49cf73fc4cf4a8ed367ab9434e75abdd65 WatchSource:0}: Error finding container e92a5219756fe1139e6a812bfaeb2c49cf73fc4cf4a8ed367ab9434e75abdd65: Status 404 returned error can't find the container with id e92a5219756fe1139e6a812bfaeb2c49cf73fc4cf4a8ed367ab9434e75abdd65
Sep 11 21:53:22.313069 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.313042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:53:22.313069 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.313061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg" event={"ID":"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5","Type":"ContainerStarted","Data":"e92a5219756fe1139e6a812bfaeb2c49cf73fc4cf4a8ed367ab9434e75abdd65"}
Sep 11 21:53:22.318897 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.318875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:53:22.451371 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-image-registry-private-configuration\") pod \"f0145b0f-57d6-475c-8517-77fb08cd5428\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") "
Sep 11 21:53:22.451542 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451381    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-trusted-ca\") pod \"f0145b0f-57d6-475c-8517-77fb08cd5428\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") "
Sep 11 21:53:22.451542 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-certificates\") pod \"f0145b0f-57d6-475c-8517-77fb08cd5428\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") "
Sep 11 21:53:22.451542 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451420    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-bound-sa-token\") pod \"f0145b0f-57d6-475c-8517-77fb08cd5428\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") "
Sep 11 21:53:22.451542 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0145b0f-57d6-475c-8517-77fb08cd5428-ca-trust-extracted\") pod \"f0145b0f-57d6-475c-8517-77fb08cd5428\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") "
Sep 11 21:53:22.451542 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vssph\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-kube-api-access-vssph\") pod \"f0145b0f-57d6-475c-8517-77fb08cd5428\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") "
Sep 11 21:53:22.451824 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451567    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-installation-pull-secrets\") pod \"f0145b0f-57d6-475c-8517-77fb08cd5428\" (UID: \"f0145b0f-57d6-475c-8517-77fb08cd5428\") "
Sep 11 21:53:22.451824 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451753    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f0145b0f-57d6-475c-8517-77fb08cd5428" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:53:22.451941 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0145b0f-57d6-475c-8517-77fb08cd5428-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f0145b0f-57d6-475c-8517-77fb08cd5428" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:22.451941 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.451884    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f0145b0f-57d6-475c-8517-77fb08cd5428" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:53:22.453846 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.453822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f0145b0f-57d6-475c-8517-77fb08cd5428" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:53:22.453938 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.453841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f0145b0f-57d6-475c-8517-77fb08cd5428" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:53:22.453938 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.453844    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f0145b0f-57d6-475c-8517-77fb08cd5428" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:53:22.454018 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.453956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-kube-api-access-vssph" (OuterVolumeSpecName: "kube-api-access-vssph") pod "f0145b0f-57d6-475c-8517-77fb08cd5428" (UID: "f0145b0f-57d6-475c-8517-77fb08cd5428"). InnerVolumeSpecName "kube-api-access-vssph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:53:22.553707 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.553457    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0145b0f-57d6-475c-8517-77fb08cd5428-ca-trust-extracted\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:53:22.553707 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.553503    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vssph\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-kube-api-access-vssph\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:53:22.553707 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.553527    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-installation-pull-secrets\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:53:22.553707 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.553550    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0145b0f-57d6-475c-8517-77fb08cd5428-image-registry-private-configuration\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:53:22.553707 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.553571    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-trusted-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:53:22.553707 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.553585    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-certificates\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:53:22.553707 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:22.553599    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-bound-sa-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:53:23.316683 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:23.316654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7455f658d9-5pblw"
Sep 11 21:53:23.345570 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:23.345510    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7455f658d9-5pblw"]
Sep 11 21:53:23.349054 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:23.348975    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7455f658d9-5pblw"]
Sep 11 21:53:23.462860 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:23.462826    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0145b0f-57d6-475c-8517-77fb08cd5428-registry-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:53:24.571266 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.571225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:53:24.571729 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.571286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:53:24.573618 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.573597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:53:24.573912 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.573893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:53:24.584867 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.584840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8cbae59-1fe9-4805-b823-dc872f45dabc-original-pull-secret\") pod \"global-pull-secret-syncer-cq4fq\" (UID: \"e8cbae59-1fe9-4805-b823-dc872f45dabc\") " pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:53:24.584985 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.584910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7aa92b-424b-400c-bce4-30a53f9f38e7-metrics-certs\") pod \"network-metrics-daemon-992bd\" (UID: \"5b7aa92b-424b-400c-bce4-30a53f9f38e7\") " pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:53:24.672162 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.672117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:53:24.674375 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.674347    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:53:24.684702 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.684678    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:53:24.695836 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.695813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnw9\" (UniqueName: \"kubernetes.io/projected/bd8ff25a-e861-4f83-8d87-c97cd89d5075-kube-api-access-hhnw9\") pod \"network-check-target-9qqhz\" (UID: \"bd8ff25a-e861-4f83-8d87-c97cd89d5075\") " pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:53:24.855482 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.855396    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0145b0f-57d6-475c-8517-77fb08cd5428" path="/var/lib/kubelet/pods/f0145b0f-57d6-475c-8517-77fb08cd5428/volumes"
Sep 11 21:53:24.864819 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.864785    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qsk4f\""
Sep 11 21:53:24.869368 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.869345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cq4fq"
Sep 11 21:53:24.873468 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.873444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-992bd"
Sep 11 21:53:24.876726 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.876702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bfmc\""
Sep 11 21:53:24.885279 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:24.885228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:53:25.076184 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.076138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:53:25.078807 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.078780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9747109c-ab47-4d2d-8297-9b8b55d19d35-metrics-tls\") pod \"dns-default-mjmp7\" (UID: \"9747109c-ab47-4d2d-8297-9b8b55d19d35\") " pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:53:25.285791 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.285760    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"]
Sep 11 21:53:25.291483 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.291457    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8zqrn\""
Sep 11 21:53:25.300365 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.300342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:53:25.307338 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.307319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"]
Sep 11 21:53:25.307429 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.307408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"
Sep 11 21:53:25.309869 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.309700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sk7jn\""
Sep 11 21:53:25.310195 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.310175    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 21:53:25.378753 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.378721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a06027be-dccb-43e0-a8aa-35971bdca8b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-q95dx\" (UID: \"a06027be-dccb-43e0-a8aa-35971bdca8b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"
Sep 11 21:53:25.479061 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.479025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a06027be-dccb-43e0-a8aa-35971bdca8b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-q95dx\" (UID: \"a06027be-dccb-43e0-a8aa-35971bdca8b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"
Sep 11 21:53:25.479210 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:25.479160    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 21:53:25.479210 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:25.479210    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06027be-dccb-43e0-a8aa-35971bdca8b9-tls-certificates podName:a06027be-dccb-43e0-a8aa-35971bdca8b9 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:25.979194484 +0000 UTC m=+67.664141176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a06027be-dccb-43e0-a8aa-35971bdca8b9-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-q95dx" (UID: "a06027be-dccb-43e0-a8aa-35971bdca8b9") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 21:53:25.646329 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.646258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-992bd"]
Sep 11 21:53:25.649204 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.649179    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mjmp7"]
Sep 11 21:53:25.659193 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.659162    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9qqhz"]
Sep 11 21:53:25.665299 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.665271    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cq4fq"]
Sep 11 21:53:25.773926 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:25.773888    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7aa92b_424b_400c_bce4_30a53f9f38e7.slice/crio-ceb73f74e089847472c588436113e6aff4b0ad0bb7c06e971a8dce8fefea7df8 WatchSource:0}: Error finding container ceb73f74e089847472c588436113e6aff4b0ad0bb7c06e971a8dce8fefea7df8: Status 404 returned error can't find the container with id ceb73f74e089847472c588436113e6aff4b0ad0bb7c06e971a8dce8fefea7df8
Sep 11 21:53:25.774905 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:25.774848    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9747109c_ab47_4d2d_8297_9b8b55d19d35.slice/crio-fbcfab92882c5ed92ec32c12fa5e8afff1c030b2a45517817d2237b5a2af6cd7 WatchSource:0}: Error finding container fbcfab92882c5ed92ec32c12fa5e8afff1c030b2a45517817d2237b5a2af6cd7: Status 404 returned error can't find the container with id fbcfab92882c5ed92ec32c12fa5e8afff1c030b2a45517817d2237b5a2af6cd7
Sep 11 21:53:25.776094 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:25.776068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8ff25a_e861_4f83_8d87_c97cd89d5075.slice/crio-b201feadad8eb17a8abdb45e16b3bb8b4fa8b7149fe98a7de7fce5bdf7c215bd WatchSource:0}: Error finding container b201feadad8eb17a8abdb45e16b3bb8b4fa8b7149fe98a7de7fce5bdf7c215bd: Status 404 returned error can't find the container with id b201feadad8eb17a8abdb45e16b3bb8b4fa8b7149fe98a7de7fce5bdf7c215bd
Sep 11 21:53:25.776783 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:25.776764    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cbae59_1fe9_4805_b823_dc872f45dabc.slice/crio-7c0ec05d364bec7956c526c357ab580cd515c0dac9bbc97a06c180459f84c3eb WatchSource:0}: Error finding container 7c0ec05d364bec7956c526c357ab580cd515c0dac9bbc97a06c180459f84c3eb: Status 404 returned error can't find the container with id 7c0ec05d364bec7956c526c357ab580cd515c0dac9bbc97a06c180459f84c3eb
Sep 11 21:53:25.983443 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.983409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a06027be-dccb-43e0-a8aa-35971bdca8b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-q95dx\" (UID: \"a06027be-dccb-43e0-a8aa-35971bdca8b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"
Sep 11 21:53:25.985747 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:25.985729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a06027be-dccb-43e0-a8aa-35971bdca8b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-q95dx\" (UID: \"a06027be-dccb-43e0-a8aa-35971bdca8b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"
Sep 11 21:53:26.216956 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.216887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"
Sep 11 21:53:26.332850 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.331905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg" event={"ID":"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5","Type":"ContainerStarted","Data":"c7a846bb3c896fa60bc8ff20c49b3d88c3079f913dc79f8d421ba1cf48311a95"}
Sep 11 21:53:26.332850 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.331976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg" event={"ID":"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5","Type":"ContainerStarted","Data":"613810a434b63e22aa1cc4d8cb77f9c86876627ec45f0a59b04a0785b79c553f"}
Sep 11 21:53:26.332850 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.331994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg" event={"ID":"a0f7a5c3-9b6a-430d-a5e7-9e75421a3dc5","Type":"ContainerStarted","Data":"21edb2852ef1f1fab61fd369732c779cfbb05a5d06dd41d941f0fd0683be61b2"}
Sep 11 21:53:26.335026 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.334909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9qqhz" event={"ID":"bd8ff25a-e861-4f83-8d87-c97cd89d5075","Type":"ContainerStarted","Data":"b201feadad8eb17a8abdb45e16b3bb8b4fa8b7149fe98a7de7fce5bdf7c215bd"}
Sep 11 21:53:26.338690 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.338662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjmp7" event={"ID":"9747109c-ab47-4d2d-8297-9b8b55d19d35","Type":"ContainerStarted","Data":"fbcfab92882c5ed92ec32c12fa5e8afff1c030b2a45517817d2237b5a2af6cd7"}
Sep 11 21:53:26.340768 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.339980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cq4fq" event={"ID":"e8cbae59-1fe9-4805-b823-dc872f45dabc","Type":"ContainerStarted","Data":"7c0ec05d364bec7956c526c357ab580cd515c0dac9bbc97a06c180459f84c3eb"}
Sep 11 21:53:26.341822 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.341733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-992bd" event={"ID":"5b7aa92b-424b-400c-bce4-30a53f9f38e7","Type":"ContainerStarted","Data":"ceb73f74e089847472c588436113e6aff4b0ad0bb7c06e971a8dce8fefea7df8"}
Sep 11 21:53:26.351112 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.350877    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dfd49fc6-dmfxg" podStartSLOduration=1.409950153 podStartE2EDuration="5.350859306s" totalImagesPullingTime="3.940909153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:26.349305483 +0000 UTC m=+68.034252202" watchObservedRunningTime="2026-09-11 21:53:26.350859306 +0000 UTC m=+68.035806022"
Sep 11 21:53:26.509141 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:26.508999    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"]
Sep 11 21:53:26.516651 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:26.516600    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06027be_dccb_43e0_a8aa_35971bdca8b9.slice/crio-9d6b67aa9abdd2f49b4981d5eae4761b783d4e378167fe3406d3d864ae05608b WatchSource:0}: Error finding container 9d6b67aa9abdd2f49b4981d5eae4761b783d4e378167fe3406d3d864ae05608b: Status 404 returned error can't find the container with id 9d6b67aa9abdd2f49b4981d5eae4761b783d4e378167fe3406d3d864ae05608b
Sep 11 21:53:27.349122 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:27.349088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx" event={"ID":"a06027be-dccb-43e0-a8aa-35971bdca8b9","Type":"ContainerStarted","Data":"9d6b67aa9abdd2f49b4981d5eae4761b783d4e378167fe3406d3d864ae05608b"}
Sep 11 21:53:28.358134 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:28.358048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-992bd" event={"ID":"5b7aa92b-424b-400c-bce4-30a53f9f38e7","Type":"ContainerStarted","Data":"0ae5f4c76b7f45e49fd6501007887dc6408b407fbebc98ca2f8264d994edee1d"}
Sep 11 21:53:31.370016 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.369977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9qqhz" event={"ID":"bd8ff25a-e861-4f83-8d87-c97cd89d5075","Type":"ContainerStarted","Data":"9e668613ec5dc85edffac4be7d19ac4c2a10068d4d955f43df35716393c71ba4"}
Sep 11 21:53:31.370477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.370115    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:53:31.371901 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.371878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjmp7" event={"ID":"9747109c-ab47-4d2d-8297-9b8b55d19d35","Type":"ContainerStarted","Data":"0d09877a4efce228b68c19f58fea6ef6ca83273f885dc6809ff62724b2817ada"}
Sep 11 21:53:31.372027 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.371919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:53:31.372027 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.371935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjmp7" event={"ID":"9747109c-ab47-4d2d-8297-9b8b55d19d35","Type":"ContainerStarted","Data":"859d2d14504fcd2c616adc1e4e54ee662d3d34ab7083b3ab5e15d57c481e4edb"}
Sep 11 21:53:31.373502 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.373480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cq4fq" event={"ID":"e8cbae59-1fe9-4805-b823-dc872f45dabc","Type":"ContainerStarted","Data":"e713098ac5a9bd5307232c950460c794c10c2328e56f4ef966dafb309599672c"}
Sep 11 21:53:31.375189 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.375162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-992bd" event={"ID":"5b7aa92b-424b-400c-bce4-30a53f9f38e7","Type":"ContainerStarted","Data":"99cc13e6b2b891c38434c657f069d1eb17052260ecb869a9fbe32849e45b8d50"}
Sep 11 21:53:31.376555 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.376538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx" event={"ID":"a06027be-dccb-43e0-a8aa-35971bdca8b9","Type":"ContainerStarted","Data":"165518fad98f0e1786d82b25c5f1c30706d28dceedf18f8ae86f2787677096f8"}
Sep 11 21:53:31.376757 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.376741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"
Sep 11 21:53:31.381382 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.381366    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx"
Sep 11 21:53:31.386273 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.386233    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9qqhz" podStartSLOduration=66.326167287 podStartE2EDuration="1m11.386219873s" totalImagesPullingTime="5.060052586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:31.385253633 +0000 UTC m=+73.070200359" watchObservedRunningTime="2026-09-11 21:53:31.386219873 +0000 UTC m=+73.071166588"
Sep 11 21:53:31.401586 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.401538    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-992bd" podStartSLOduration=69.954224928 podStartE2EDuration="1m11.401523974s" totalImagesPullingTime="1.447299046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:31.400783803 +0000 UTC m=+73.085730509" watchObservedRunningTime="2026-09-11 21:53:31.401523974 +0000 UTC m=+73.086470681"
Sep 11 21:53:31.414698 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.414628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cq4fq" podStartSLOduration=66.331461988 podStartE2EDuration="1m11.414610061s" totalImagesPullingTime="5.083148073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:31.414471282 +0000 UTC m=+73.099417996" watchObservedRunningTime="2026-09-11 21:53:31.414610061 +0000 UTC m=+73.099556776"
Sep 11 21:53:31.430588 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.430521    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mjmp7" podStartSLOduration=33.406120947 podStartE2EDuration="38.430506304s" totalImagesPullingTime="5.024385357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:31.429567994 +0000 UTC m=+73.114514705" watchObservedRunningTime="2026-09-11 21:53:31.430506304 +0000 UTC m=+73.115453018"
Sep 11 21:53:31.446015 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:31.445960    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-q95dx" podStartSLOduration=2.135090708 podStartE2EDuration="6.445940918s" totalImagesPullingTime="4.31085021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:31.444525088 +0000 UTC m=+73.129471803" watchObservedRunningTime="2026-09-11 21:53:31.445940918 +0000 UTC m=+73.130887631"
Sep 11 21:53:37.718181 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.718140    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"]
Sep 11 21:53:37.723083 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.723060    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m2mw7"]
Sep 11 21:53:37.723230 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.723179    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.725951 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.725918    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 21:53:37.725951 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.725944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 21:53:37.726229 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.726209    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9wlf2\""
Sep 11 21:53:37.726379 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.726363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.726825 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.726794    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 21:53:37.726825 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.726817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 21:53:37.726965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.726826    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 21:53:37.726965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.726848    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 21:53:37.728569 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.728554    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b2br2\""
Sep 11 21:53:37.728759 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.728747    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 21:53:37.729008 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.728991    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 21:53:37.729077 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.729026    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 21:53:37.733511 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.733489    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"]
Sep 11 21:53:37.769742 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.769706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6nc\" (UniqueName: \"kubernetes.io/projected/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-api-access-kn6nc\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.769928 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.769749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.769928 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.769783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-textfile\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.769928 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.769806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.769928 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.769834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-sys\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.769928 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.769856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.770100 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.769970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61e0d08f-9078-4087-bf40-fc0457d37a8d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.770100 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.770000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6scp7\" (UniqueName: \"kubernetes.io/projected/b88eb3f2-f059-4de3-8403-963ca05463c5-kube-api-access-6scp7\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.770100 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.770034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-tls\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.770100 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.770057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.770100 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.770075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-wtmp\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.770100 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.770092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e0d08f-9078-4087-bf40-fc0457d37a8d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.770290 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.770120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-root\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.770290 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.770159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.770290 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.770198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88eb3f2-f059-4de3-8403-963ca05463c5-metrics-client-ca\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.871245 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-sys\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.871245 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61e0d08f-9078-4087-bf40-fc0457d37a8d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6scp7\" (UniqueName: \"kubernetes.io/projected/b88eb3f2-f059-4de3-8403-963ca05463c5-kube-api-access-6scp7\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-tls\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-sys\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-wtmp\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e0d08f-9078-4087-bf40-fc0457d37a8d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-root\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:37.871421    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88eb3f2-f059-4de3-8403-963ca05463c5-metrics-client-ca\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.871495 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:37.871490    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-tls podName:b88eb3f2-f059-4de3-8403-963ca05463c5 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:38.371463352 +0000 UTC m=+80.056410061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-tls") pod "node-exporter-m2mw7" (UID: "b88eb3f2-f059-4de3-8403-963ca05463c5") : secret "node-exporter-tls" not found
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-root\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6nc\" (UniqueName: \"kubernetes.io/projected/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-api-access-kn6nc\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-wtmp\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:37.871608    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-textfile\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:37.871689    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-tls podName:61e0d08f-9078-4087-bf40-fc0457d37a8d nodeName:}" failed. No retries permitted until 2026-09-11 21:53:38.371668941 +0000 UTC m=+80.056615649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-rf89z" (UID: "61e0d08f-9078-4087-bf40-fc0457d37a8d") : secret "kube-state-metrics-tls" not found
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61e0d08f-9078-4087-bf40-fc0457d37a8d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.872176 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.871965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-textfile\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.872518 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.872352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.872518 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.872368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.872518 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.872391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88eb3f2-f059-4de3-8403-963ca05463c5-metrics-client-ca\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.872853 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.872837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e0d08f-9078-4087-bf40-fc0457d37a8d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.873719 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.873694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:37.874031 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.874010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.880045 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.880003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6scp7\" (UniqueName: \"kubernetes.io/projected/b88eb3f2-f059-4de3-8403-963ca05463c5-kube-api-access-6scp7\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:37.884044 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:37.884022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6nc\" (UniqueName: \"kubernetes.io/projected/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-api-access-kn6nc\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:38.376929 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.376892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-tls\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:38.376929 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.376939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:38.379278 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.379248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b88eb3f2-f059-4de3-8403-963ca05463c5-node-exporter-tls\") pod \"node-exporter-m2mw7\" (UID: \"b88eb3f2-f059-4de3-8403-963ca05463c5\") " pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:38.379386 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.379259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61e0d08f-9078-4087-bf40-fc0457d37a8d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rf89z\" (UID: \"61e0d08f-9078-4087-bf40-fc0457d37a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:38.634436 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.634354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"
Sep 11 21:53:38.639682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.639666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m2mw7"
Sep 11 21:53:38.649868 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:38.649843    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88eb3f2_f059_4de3_8403_963ca05463c5.slice/crio-d936a768f0029067b9b9934029c6a1158cd73b06d96b3535d76b1a6ed9cb87b8 WatchSource:0}: Error finding container d936a768f0029067b9b9934029c6a1158cd73b06d96b3535d76b1a6ed9cb87b8: Status 404 returned error can't find the container with id d936a768f0029067b9b9934029c6a1158cd73b06d96b3535d76b1a6ed9cb87b8
Sep 11 21:53:38.804910 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.804875    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:53:38.809951 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.809923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.812175 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812138    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 21:53:38.812306 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812142    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 21:53:38.812306 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812270    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 21:53:38.812424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812325    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 21:53:38.812424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812330    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 21:53:38.812512 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 21:53:38.812512 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812503    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 21:53:38.812600 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812579    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 21:53:38.812797 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.812784    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6mw9m\""
Sep 11 21:53:38.813055 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.813043    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 21:53:38.825372 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.825345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:53:38.881485 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2pf\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-kube-api-access-sg2pf\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-out\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-web-config\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881682 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881980 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881980 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881980 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.881980 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.881866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.906771 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.906696    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z"]
Sep 11 21:53:38.909851 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:38.909817    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e0d08f_9078_4087_bf40_fc0457d37a8d.slice/crio-655313c2d111e61b403fd76eee87edf258d53440bb143636603f4f5d120e8e09 WatchSource:0}: Error finding container 655313c2d111e61b403fd76eee87edf258d53440bb143636603f4f5d120e8e09: Status 404 returned error can't find the container with id 655313c2d111e61b403fd76eee87edf258d53440bb143636603f4f5d120e8e09
Sep 11 21:53:38.983046 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983046 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983264 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983264 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983340 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983386 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2pf\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-kube-api-access-sg2pf\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983386 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-out\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-web-config\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983477 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983613 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.983613 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.984195 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.983609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.984195 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:38.983785    2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 11 21:53:38.984195 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:38.983851    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls podName:a60de666-385b-42a3-9ff8-0a45a582b7a1 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:39.483830974 +0000 UTC m=+81.168777683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1") : secret "alertmanager-main-tls" not found
Sep 11 21:53:38.984195 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:53:38.984149    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle podName:a60de666-385b-42a3-9ff8-0a45a582b7a1 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:39.484130484 +0000 UTC m=+81.169077444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1") : configmap references non-existent config key: ca-bundle.crt
Sep 11 21:53:38.984483 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.984456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.984929 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.984902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.986403 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.986377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.986516 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.986499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.986569 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.986503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.986819 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.986798    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.986902 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.986872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-web-config\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.987038 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.987018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.987175 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.987159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.987318 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.987296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-out\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:38.994393 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:38.994375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2pf\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-kube-api-access-sg2pf\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:39.409271 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:39.409225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2mw7" event={"ID":"b88eb3f2-f059-4de3-8403-963ca05463c5","Type":"ContainerStarted","Data":"d936a768f0029067b9b9934029c6a1158cd73b06d96b3535d76b1a6ed9cb87b8"}
Sep 11 21:53:39.410890 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:39.410864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z" event={"ID":"61e0d08f-9078-4087-bf40-fc0457d37a8d","Type":"ContainerStarted","Data":"655313c2d111e61b403fd76eee87edf258d53440bb143636603f4f5d120e8e09"}
Sep 11 21:53:39.487620 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:39.487548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:39.487830 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:39.487652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:39.488610 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:39.488561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:39.490766 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:39.490677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:39.719842 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:39.719809    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:53:40.293302 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:40.293209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:53:40.299848 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:53:40.299812    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60de666_385b_42a3_9ff8_0a45a582b7a1.slice/crio-6f65fbc5d681a6c8d9e6d5e710e8b98999ee766f1cb3425832988dd07bd85160 WatchSource:0}: Error finding container 6f65fbc5d681a6c8d9e6d5e710e8b98999ee766f1cb3425832988dd07bd85160: Status 404 returned error can't find the container with id 6f65fbc5d681a6c8d9e6d5e710e8b98999ee766f1cb3425832988dd07bd85160
Sep 11 21:53:40.416461 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:40.416417    2578 generic.go:356] "Generic (PLEG): container finished" podID="b88eb3f2-f059-4de3-8403-963ca05463c5" containerID="59221af7296c505b2169d8b97cc3edda5ace796918881316081a7e72bb0938fe" exitCode=0
Sep 11 21:53:40.416669 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:40.416506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2mw7" event={"ID":"b88eb3f2-f059-4de3-8403-963ca05463c5","Type":"ContainerDied","Data":"59221af7296c505b2169d8b97cc3edda5ace796918881316081a7e72bb0938fe"}
Sep 11 21:53:40.419037 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:40.419010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z" event={"ID":"61e0d08f-9078-4087-bf40-fc0457d37a8d","Type":"ContainerStarted","Data":"0a1c3845ada3464292dea8fa712d7266f8c3ca1e7ca9c3544a1e94b2fe638605"}
Sep 11 21:53:40.419164 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:40.419066    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z" event={"ID":"61e0d08f-9078-4087-bf40-fc0457d37a8d","Type":"ContainerStarted","Data":"0b41d8f5787696af36378262b61db5902d6e063e1de277ac1de0b0eaf06b314d"}
Sep 11 21:53:40.419164 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:40.419098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z" event={"ID":"61e0d08f-9078-4087-bf40-fc0457d37a8d","Type":"ContainerStarted","Data":"7c834d533041072107b9dd56e43fa15d42e1ef9aae2ec42021379832ad7f7f5c"}
Sep 11 21:53:40.420290 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:40.420270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerStarted","Data":"6f65fbc5d681a6c8d9e6d5e710e8b98999ee766f1cb3425832988dd07bd85160"}
Sep 11 21:53:40.454359 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:40.454309    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rf89z" podStartSLOduration=2.295527381 podStartE2EDuration="3.454292309s" totalImagesPullingTime="1.158764928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:40.453112838 +0000 UTC m=+82.138059551" watchObservedRunningTime="2026-09-11 21:53:40.454292309 +0000 UTC m=+82.139239022"
Sep 11 21:53:41.381737 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:41.381704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mjmp7"
Sep 11 21:53:41.427282 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:41.427248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2mw7" event={"ID":"b88eb3f2-f059-4de3-8403-963ca05463c5","Type":"ContainerStarted","Data":"b316d58e167f4d8432f5322e27c3cf65890df548c93a86ede6a0066c876a1381"}
Sep 11 21:53:41.427403 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:41.427316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2mw7" event={"ID":"b88eb3f2-f059-4de3-8403-963ca05463c5","Type":"ContainerStarted","Data":"365bd4864ee3ec69a7bec73df549ae85056f7c6e2391b6e81a1bae2a16c7855e"}
Sep 11 21:53:41.448316 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:41.448249    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m2mw7" podStartSLOduration=3.474139065 podStartE2EDuration="4.448228629s" totalImagesPullingTime="905.414796ms" totalInitContainerRuntime="68.674768ms" isStatefulPod=true podCreationTimestamp="2026-09-11 21:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:41.447521039 +0000 UTC m=+83.132467766" watchObservedRunningTime="2026-09-11 21:53:41.448228629 +0000 UTC m=+83.133175343"
Sep 11 21:53:42.436923 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:42.436891    2578 generic.go:356] "Generic (PLEG): container finished" podID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerID="c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546" exitCode=0
Sep 11 21:53:42.437334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:42.436969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerDied","Data":"c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546"}
Sep 11 21:53:44.448413 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:44.448377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerStarted","Data":"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a"}
Sep 11 21:53:44.448821 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:44.448449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerStarted","Data":"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87"}
Sep 11 21:53:44.448821 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:44.448477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerStarted","Data":"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70"}
Sep 11 21:53:44.448821 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:44.448514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerStarted","Data":"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21"}
Sep 11 21:53:44.448821 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:44.448543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerStarted","Data":"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4"}
Sep 11 21:53:45.455307 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:45.455265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerStarted","Data":"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3"}
Sep 11 21:53:45.487923 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:53:45.487869    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.538732618 podStartE2EDuration="7.487855981s" totalImagesPullingTime="2.934463893s" totalInitContainerRuntime="14.65947ms" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:45.484501156 +0000 UTC m=+87.169447864" watchObservedRunningTime="2026-09-11 21:53:45.487855981 +0000 UTC m=+87.172802766"
Sep 11 21:54:02.382263 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:02.382224    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9qqhz"
Sep 11 21:54:58.065539 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.065497    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:54:58.066102 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.065945    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="alertmanager" containerID="cri-o://1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4" gracePeriod=120
Sep 11 21:54:58.066102 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.065973    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy-metric" containerID="cri-o://4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a" gracePeriod=120
Sep 11 21:54:58.066102 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.066002    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="prom-label-proxy" containerID="cri-o://d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3" gracePeriod=120
Sep 11 21:54:58.066102 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.066026    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy-web" containerID="cri-o://98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70" gracePeriod=120
Sep 11 21:54:58.066102 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.066041    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="config-reloader" containerID="cri-o://9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21" gracePeriod=120
Sep 11 21:54:58.066102 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.066049    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy" containerID="cri-o://1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87" gracePeriod=120
Sep 11 21:54:58.717844 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.717812    2578 generic.go:356] "Generic (PLEG): container finished" podID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerID="d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3" exitCode=0
Sep 11 21:54:58.717844 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.717839    2578 generic.go:356] "Generic (PLEG): container finished" podID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerID="1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87" exitCode=0
Sep 11 21:54:58.717844 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.717845    2578 generic.go:356] "Generic (PLEG): container finished" podID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerID="9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21" exitCode=0
Sep 11 21:54:58.717844 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.717852    2578 generic.go:356] "Generic (PLEG): container finished" podID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerID="1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4" exitCode=0
Sep 11 21:54:58.718157 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.717884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerDied","Data":"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3"}
Sep 11 21:54:58.718157 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.717964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerDied","Data":"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87"}
Sep 11 21:54:58.718157 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.718003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerDied","Data":"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21"}
Sep 11 21:54:58.718157 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:58.718039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerDied","Data":"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4"}
Sep 11 21:54:59.315980 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.315957    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.378643 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.378813 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg2pf\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-kube-api-access-sg2pf\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.378813 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.378813 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378727    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-tls-assets\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.378813 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378744    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-main-db\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.378813 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.378813 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378787    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-volume\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.379168 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378840    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-out\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.379168 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-web-config\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.379168 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378924    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.379168 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-cluster-tls-config\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.379168 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.378983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-metrics-client-ca\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.379168 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.379008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a60de666-385b-42a3-9ff8-0a45a582b7a1\" (UID: \"a60de666-385b-42a3-9ff8-0a45a582b7a1\") "
Sep 11 21:54:59.379168 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.379115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:54:59.379168 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.379160    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:59.379571 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.379241    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-main-db\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.379571 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.379260    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.379701 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.379587    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:59.381924 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.381778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:59.381924 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.381867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-kube-api-access-sg2pf" (OuterVolumeSpecName: "kube-api-access-sg2pf") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "kube-api-access-sg2pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:54:59.381924 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.381898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:54:59.382532 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.382503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-out" (OuterVolumeSpecName: "config-out") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:54:59.382678 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.382603    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:59.382749 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.382669    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-volume" (OuterVolumeSpecName: "config-volume") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:59.383451 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.383425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:59.383901 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.383870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:59.386522 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.386489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:59.393432 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.393381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-web-config" (OuterVolumeSpecName: "web-config") pod "a60de666-385b-42a3-9ff8-0a45a582b7a1" (UID: "a60de666-385b-42a3-9ff8-0a45a582b7a1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:59.479850 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479814    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-out\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.479850 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479846    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-web-config\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479859    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-main-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479873    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-cluster-tls-config\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479885    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60de666-385b-42a3-9ff8-0a45a582b7a1-metrics-client-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479899    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479911    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479923    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg2pf\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-kube-api-access-sg2pf\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479935    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a60de666-385b-42a3-9ff8-0a45a582b7a1-tls-assets\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479946    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.480062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.479957    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a60de666-385b-42a3-9ff8-0a45a582b7a1-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:54:59.724762 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.724726    2578 generic.go:356] "Generic (PLEG): container finished" podID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerID="4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a" exitCode=0
Sep 11 21:54:59.724762 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.724751    2578 generic.go:356] "Generic (PLEG): container finished" podID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerID="98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70" exitCode=0
Sep 11 21:54:59.724965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.724794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerDied","Data":"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a"}
Sep 11 21:54:59.724965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.724823    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.724965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.724907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerDied","Data":"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70"}
Sep 11 21:54:59.724965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.724935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a60de666-385b-42a3-9ff8-0a45a582b7a1","Type":"ContainerDied","Data":"6f65fbc5d681a6c8d9e6d5e710e8b98999ee766f1cb3425832988dd07bd85160"}
Sep 11 21:54:59.724965 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.724965    2578 scope.go:117] "RemoveContainer" containerID="d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3"
Sep 11 21:54:59.733521 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.733505    2578 scope.go:117] "RemoveContainer" containerID="4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a"
Sep 11 21:54:59.740319 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.740301    2578 scope.go:117] "RemoveContainer" containerID="1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87"
Sep 11 21:54:59.746760 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.746741    2578 scope.go:117] "RemoveContainer" containerID="98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70"
Sep 11 21:54:59.748942 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.748916    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:54:59.752747 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.752694    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:54:59.754347 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.754325    2578 scope.go:117] "RemoveContainer" containerID="9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21"
Sep 11 21:54:59.761428 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.761123    2578 scope.go:117] "RemoveContainer" containerID="1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4"
Sep 11 21:54:59.768067 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.768050    2578 scope.go:117] "RemoveContainer" containerID="c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546"
Sep 11 21:54:59.774680 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.774628    2578 scope.go:117] "RemoveContainer" containerID="d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3"
Sep 11 21:54:59.774972 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:54:59.774949    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3\": container with ID starting with d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3 not found: ID does not exist" containerID="d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3"
Sep 11 21:54:59.775048 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.774987    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3"} err="failed to get container status \"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3\": rpc error: code = NotFound desc = could not find container \"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3\": container with ID starting with d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3 not found: ID does not exist"
Sep 11 21:54:59.775048 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.775035    2578 scope.go:117] "RemoveContainer" containerID="4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a"
Sep 11 21:54:59.775458 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:54:59.775408    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a\": container with ID starting with 4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a not found: ID does not exist" containerID="4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a"
Sep 11 21:54:59.775458 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.775441    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a"} err="failed to get container status \"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a\": rpc error: code = NotFound desc = could not find container \"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a\": container with ID starting with 4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a not found: ID does not exist"
Sep 11 21:54:59.775687 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.775467    2578 scope.go:117] "RemoveContainer" containerID="1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87"
Sep 11 21:54:59.775869 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:54:59.775833    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87\": container with ID starting with 1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87 not found: ID does not exist" containerID="1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87"
Sep 11 21:54:59.775948 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.775870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87"} err="failed to get container status \"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87\": rpc error: code = NotFound desc = could not find container \"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87\": container with ID starting with 1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87 not found: ID does not exist"
Sep 11 21:54:59.775948 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.775896    2578 scope.go:117] "RemoveContainer" containerID="98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70"
Sep 11 21:54:59.776254 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:54:59.776224    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70\": container with ID starting with 98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70 not found: ID does not exist" containerID="98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70"
Sep 11 21:54:59.776357 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.776261    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70"} err="failed to get container status \"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70\": rpc error: code = NotFound desc = could not find container \"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70\": container with ID starting with 98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70 not found: ID does not exist"
Sep 11 21:54:59.776357 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.776283    2578 scope.go:117] "RemoveContainer" containerID="9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21"
Sep 11 21:54:59.776588 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:54:59.776570    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21\": container with ID starting with 9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21 not found: ID does not exist" containerID="9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21"
Sep 11 21:54:59.776659 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.776593    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21"} err="failed to get container status \"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21\": rpc error: code = NotFound desc = could not find container \"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21\": container with ID starting with 9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21 not found: ID does not exist"
Sep 11 21:54:59.776659 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.776609    2578 scope.go:117] "RemoveContainer" containerID="1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4"
Sep 11 21:54:59.776916 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:54:59.776895    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4\": container with ID starting with 1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4 not found: ID does not exist" containerID="1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4"
Sep 11 21:54:59.776960 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.776924    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4"} err="failed to get container status \"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4\": rpc error: code = NotFound desc = could not find container \"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4\": container with ID starting with 1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4 not found: ID does not exist"
Sep 11 21:54:59.776960 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.776947    2578 scope.go:117] "RemoveContainer" containerID="c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546"
Sep 11 21:54:59.777063 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777048    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:54:59.777227 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:54:59.777210    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546\": container with ID starting with c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546 not found: ID does not exist" containerID="c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546"
Sep 11 21:54:59.777276 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777231    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546"} err="failed to get container status \"c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546\": rpc error: code = NotFound desc = could not find container \"c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546\": container with ID starting with c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546 not found: ID does not exist"
Sep 11 21:54:59.777276 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777245    2578 scope.go:117] "RemoveContainer" containerID="d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777288    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="config-reloader"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777300    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="config-reloader"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777308    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777313    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777320    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="alertmanager"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777327    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="alertmanager"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777333    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy-web"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777341    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy-web"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777350    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy-metric"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777358    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy-metric"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777373    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="init-config-reloader"
Sep 11 21:54:59.777376 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777378    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="init-config-reloader"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777387    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="prom-label-proxy"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777395    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="prom-label-proxy"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777454    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy-metric"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777464    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="config-reloader"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777463    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3"} err="failed to get container status \"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3\": rpc error: code = NotFound desc = could not find container \"d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3\": container with ID starting with d5988afc91426eead63c77a777da26520a87392faa7e9d0312522c381e12f2c3 not found: ID does not exist"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777490    2578 scope.go:117] "RemoveContainer" containerID="4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777469    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="alertmanager"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777537    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777547    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="kube-rbac-proxy-web"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777557    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" containerName="prom-label-proxy"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777728    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a"} err="failed to get container status \"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a\": rpc error: code = NotFound desc = could not find container \"4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a\": container with ID starting with 4ee73354388796a35c18f4c9f50f25c46e5b2da1d0e7b9570bec4cd1cceb000a not found: ID does not exist"
Sep 11 21:54:59.777769 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.777745    2578 scope.go:117] "RemoveContainer" containerID="1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87"
Sep 11 21:54:59.778195 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778003    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87"} err="failed to get container status \"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87\": rpc error: code = NotFound desc = could not find container \"1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87\": container with ID starting with 1c709f6d52fce074011c3544f6985cdf4db7c252f4554005d67843e13642cb87 not found: ID does not exist"
Sep 11 21:54:59.778195 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778018    2578 scope.go:117] "RemoveContainer" containerID="98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70"
Sep 11 21:54:59.778258 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778228    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70"} err="failed to get container status \"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70\": rpc error: code = NotFound desc = could not find container \"98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70\": container with ID starting with 98822adc812c977400cd4d334ff628a920d8319a8bdef37556bebb9a3f106d70 not found: ID does not exist"
Sep 11 21:54:59.778258 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778241    2578 scope.go:117] "RemoveContainer" containerID="9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21"
Sep 11 21:54:59.778495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778475    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21"} err="failed to get container status \"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21\": rpc error: code = NotFound desc = could not find container \"9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21\": container with ID starting with 9f12f50a4049ba747021cf4256eda2e82f0359b1a5e582b8ceaa70b922df3f21 not found: ID does not exist"
Sep 11 21:54:59.778495 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778494    2578 scope.go:117] "RemoveContainer" containerID="1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4"
Sep 11 21:54:59.778776 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778755    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4"} err="failed to get container status \"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4\": rpc error: code = NotFound desc = could not find container \"1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4\": container with ID starting with 1b387eb8dbf50bd9c434f0b64dfe55a90b9b9bc4cd9cf44d456813091c1487a4 not found: ID does not exist"
Sep 11 21:54:59.778776 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778774    2578 scope.go:117] "RemoveContainer" containerID="c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546"
Sep 11 21:54:59.779017 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.778998    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546"} err="failed to get container status \"c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546\": rpc error: code = NotFound desc = could not find container \"c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546\": container with ID starting with c3162cc61c77054a9b137d1ca2a9b7170dc370c1754f2c05463ea29e8ff64546 not found: ID does not exist"
Sep 11 21:54:59.782284 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.782264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.784245 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784227    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 21:54:59.784346 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784262    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6mw9m\""
Sep 11 21:54:59.784588 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 21:54:59.784696 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 21:54:59.784946 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 21:54:59.784946 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 21:54:59.785073 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 21:54:59.785073 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784976    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 21:54:59.785073 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.784981    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 21:54:59.790047 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.790026    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 21:54:59.794029 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.794005    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:54:59.883220 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b49caf-7abf-4067-a6cc-366ae9016753-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883220 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65b49caf-7abf-4067-a6cc-366ae9016753-config-out\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65b49caf-7abf-4067-a6cc-366ae9016753-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7pt5\" (UniqueName: \"kubernetes.io/projected/65b49caf-7abf-4067-a6cc-366ae9016753-kube-api-access-w7pt5\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65b49caf-7abf-4067-a6cc-366ae9016753-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65b49caf-7abf-4067-a6cc-366ae9016753-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883424 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883660 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883660 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883660 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-config-volume\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.883660 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.883559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-web-config\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984621 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7pt5\" (UniqueName: \"kubernetes.io/projected/65b49caf-7abf-4067-a6cc-366ae9016753-kube-api-access-w7pt5\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984621 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65b49caf-7abf-4067-a6cc-366ae9016753-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984621 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65b49caf-7abf-4067-a6cc-366ae9016753-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984882 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984882 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984882 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984882 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-config-volume\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984882 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-web-config\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.984882 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b49caf-7abf-4067-a6cc-366ae9016753-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.985053 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65b49caf-7abf-4067-a6cc-366ae9016753-config-out\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.985053 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65b49caf-7abf-4067-a6cc-366ae9016753-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.985053 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.985053 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65b49caf-7abf-4067-a6cc-366ae9016753-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.985053 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.984995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.986018 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.985796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65b49caf-7abf-4067-a6cc-366ae9016753-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.986198 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.986066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b49caf-7abf-4067-a6cc-366ae9016753-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.988327 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.988156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65b49caf-7abf-4067-a6cc-366ae9016753-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.988327 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.988229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65b49caf-7abf-4067-a6cc-366ae9016753-config-out\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.988327 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.988265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.988528 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.988336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-config-volume\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.988528 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.988383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.988528 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.988382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.988528 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.988450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.988700 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.988584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-web-config\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.990094 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.990070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65b49caf-7abf-4067-a6cc-366ae9016753-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:54:59.991403 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:54:59.991384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7pt5\" (UniqueName: \"kubernetes.io/projected/65b49caf-7abf-4067-a6cc-366ae9016753-kube-api-access-w7pt5\") pod \"alertmanager-main-0\" (UID: \"65b49caf-7abf-4067-a6cc-366ae9016753\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:55:00.091847 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:00.091793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 21:55:00.369878 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:00.369512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 21:55:00.372170 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:55:00.372145    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b49caf_7abf_4067_a6cc_366ae9016753.slice/crio-6c79b93fc58764e5c04c63583b3cd792888ab45817490332d0921c9866c3163d WatchSource:0}: Error finding container 6c79b93fc58764e5c04c63583b3cd792888ab45817490332d0921c9866c3163d: Status 404 returned error can't find the container with id 6c79b93fc58764e5c04c63583b3cd792888ab45817490332d0921c9866c3163d
Sep 11 21:55:00.731248 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:00.731210    2578 generic.go:356] "Generic (PLEG): container finished" podID="65b49caf-7abf-4067-a6cc-366ae9016753" containerID="7f82bd426e49e154b6d65a6f63c0e20dc14ae2e3070b4e8b1ffc3ce3ba167fc9" exitCode=0
Sep 11 21:55:00.731248 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:00.731247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65b49caf-7abf-4067-a6cc-366ae9016753","Type":"ContainerDied","Data":"7f82bd426e49e154b6d65a6f63c0e20dc14ae2e3070b4e8b1ffc3ce3ba167fc9"}
Sep 11 21:55:00.731509 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:00.731297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65b49caf-7abf-4067-a6cc-366ae9016753","Type":"ContainerStarted","Data":"6c79b93fc58764e5c04c63583b3cd792888ab45817490332d0921c9866c3163d"}
Sep 11 21:55:00.862935 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:00.862907    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60de666-385b-42a3-9ff8-0a45a582b7a1" path="/var/lib/kubelet/pods/a60de666-385b-42a3-9ff8-0a45a582b7a1/volumes"
Sep 11 21:55:01.738591 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:01.738556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65b49caf-7abf-4067-a6cc-366ae9016753","Type":"ContainerStarted","Data":"740aa0aa350f86d31221b89cf6a56e43d114562818e49863211b31239eb8abf9"}
Sep 11 21:55:01.738963 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:01.738623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65b49caf-7abf-4067-a6cc-366ae9016753","Type":"ContainerStarted","Data":"f5d4fb89f824805a49e38c45df3be69caaebfba8b28852e1ff147ae79df42394"}
Sep 11 21:55:01.738963 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:01.738671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65b49caf-7abf-4067-a6cc-366ae9016753","Type":"ContainerStarted","Data":"880be2edd4eaa88ea456011b38b6eec35d7797a2f850bfa553692c4160afda5d"}
Sep 11 21:55:01.738963 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:01.738695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65b49caf-7abf-4067-a6cc-366ae9016753","Type":"ContainerStarted","Data":"4891ff9c53771dd9e76d9f8ebafa1379a08b984f0a387b563cf76e0cb6c37c71"}
Sep 11 21:55:01.738963 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:01.738719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65b49caf-7abf-4067-a6cc-366ae9016753","Type":"ContainerStarted","Data":"6e4e20701b879b6751726ccf897d95f52decfdd55e0a3b4ba68c39839968ea64"}
Sep 11 21:55:01.738963 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:01.738743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65b49caf-7abf-4067-a6cc-366ae9016753","Type":"ContainerStarted","Data":"aa1ba0146b86a61cfb1bea9b46fa08a8df5009a2776b9336fa826ee7522e0759"}
Sep 11 21:55:01.764584 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:01.764529    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.747513627 podStartE2EDuration="2.764512523s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.998896ms" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:01.763365518 +0000 UTC m=+163.448312243" watchObservedRunningTime="2026-09-11 21:55:01.764512523 +0000 UTC m=+163.449459307"
Sep 11 21:55:34.613269 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.613230    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"]
Sep 11 21:55:34.617961 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.617933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.620171 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.620143    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 21:55:34.620501 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.620482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9nhgg\""
Sep 11 21:55:34.620829 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.620814    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 21:55:34.624655 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.624599    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"]
Sep 11 21:55:34.755585 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.755544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294tm\" (UniqueName: \"kubernetes.io/projected/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-kube-api-access-294tm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.755822 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.755598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.755822 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.755716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.856847 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.856812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.856847 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.856850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.857084 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.856889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294tm\" (UniqueName: \"kubernetes.io/projected/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-kube-api-access-294tm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.857237 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.857215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.857237 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.857228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.864651 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.864581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-294tm\" (UniqueName: \"kubernetes.io/projected/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-kube-api-access-294tm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:34.927269 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:34.927231    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:35.194791 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:35.194753    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"]
Sep 11 21:55:35.197986 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:55:35.197956    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8feb7f_e7c0_4797_8fb2_6cccb23ac8c9.slice/crio-d480a8b4146f9e92b8e16cb063d44902e9bdc09f3b71383ab8d9ed5a201b1e9d WatchSource:0}: Error finding container d480a8b4146f9e92b8e16cb063d44902e9bdc09f3b71383ab8d9ed5a201b1e9d: Status 404 returned error can't find the container with id d480a8b4146f9e92b8e16cb063d44902e9bdc09f3b71383ab8d9ed5a201b1e9d
Sep 11 21:55:35.859555 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:35.859516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv" event={"ID":"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9","Type":"ContainerStarted","Data":"d480a8b4146f9e92b8e16cb063d44902e9bdc09f3b71383ab8d9ed5a201b1e9d"}
Sep 11 21:55:41.882872 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:41.882775    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerID="4fc74fdb6bee21020dd5567688f644d0c0c13495deebd7363a790be6bf2a0e56" exitCode=0
Sep 11 21:55:41.883223 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:41.882864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv" event={"ID":"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9","Type":"ContainerDied","Data":"4fc74fdb6bee21020dd5567688f644d0c0c13495deebd7363a790be6bf2a0e56"}
Sep 11 21:55:44.895270 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:44.895236    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerID="0d2392c070172118728728f10b526d85738a7d68432cab475175b79bac786373" exitCode=0
Sep 11 21:55:44.895722 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:44.895326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv" event={"ID":"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9","Type":"ContainerDied","Data":"0d2392c070172118728728f10b526d85738a7d68432cab475175b79bac786373"}
Sep 11 21:55:51.931725 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:51.931685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv" event={"ID":"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9","Type":"ContainerStarted","Data":"3f985d3f43dc8b29566054034358c0d2a5fd3e9854f2e77b61baba271bd81373"}
Sep 11 21:55:51.948220 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:51.948163    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv" podStartSLOduration=2.6349644359999997 podStartE2EDuration="17.948147539s" totalImagesPullingTime="15.313183103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:51.946931556 +0000 UTC m=+213.631878272" watchObservedRunningTime="2026-09-11 21:55:51.948147539 +0000 UTC m=+213.633094252"
Sep 11 21:55:52.937385 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:52.937340    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerID="3f985d3f43dc8b29566054034358c0d2a5fd3e9854f2e77b61baba271bd81373" exitCode=0
Sep 11 21:55:52.937762 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:52.937426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv" event={"ID":"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9","Type":"ContainerDied","Data":"3f985d3f43dc8b29566054034358c0d2a5fd3e9854f2e77b61baba271bd81373"}
Sep 11 21:55:54.063573 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.063551    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:54.211658 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.211550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-util\") pod \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") "
Sep 11 21:55:54.211658 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.211601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-bundle\") pod \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") "
Sep 11 21:55:54.211864 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.211680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-294tm\" (UniqueName: \"kubernetes.io/projected/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-kube-api-access-294tm\") pod \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\" (UID: \"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9\") "
Sep 11 21:55:54.212436 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.212405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-bundle" (OuterVolumeSpecName: "bundle") pod "8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" (UID: "8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:54.213831 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.213804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-kube-api-access-294tm" (OuterVolumeSpecName: "kube-api-access-294tm") pod "8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" (UID: "8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9"). InnerVolumeSpecName "kube-api-access-294tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:55:54.216406 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.216383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-util" (OuterVolumeSpecName: "util") pod "8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" (UID: "8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:54.312860 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.312814    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:55:54.312860 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.312858    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-294tm\" (UniqueName: \"kubernetes.io/projected/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-kube-api-access-294tm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:55:54.312860 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.312868    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9-util\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:55:54.946399 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.946359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv" event={"ID":"8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9","Type":"ContainerDied","Data":"d480a8b4146f9e92b8e16cb063d44902e9bdc09f3b71383ab8d9ed5a201b1e9d"}
Sep 11 21:55:54.946399 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.946399    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pglwv"
Sep 11 21:55:54.946399 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:55:54.946408    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d480a8b4146f9e92b8e16cb063d44902e9bdc09f3b71383ab8d9ed5a201b1e9d"
Sep 11 21:56:02.790941 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.790905    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"]
Sep 11 21:56:02.791330 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.791180    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerName="pull"
Sep 11 21:56:02.791330 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.791195    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerName="pull"
Sep 11 21:56:02.791330 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.791209    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerName="extract"
Sep 11 21:56:02.791330 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.791214    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerName="extract"
Sep 11 21:56:02.791330 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.791228    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerName="util"
Sep 11 21:56:02.791330 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.791233    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerName="util"
Sep 11 21:56:02.791330 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.791283    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8feb7f-e7c0-4797-8fb2-6cccb23ac8c9" containerName="extract"
Sep 11 21:56:02.794474 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.794453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"
Sep 11 21:56:02.796438 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.796420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-cs4gf\""
Sep 11 21:56:02.796824 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.796804    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 21:56:02.797142 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.797126    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:56:02.811048 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.811018    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"]
Sep 11 21:56:02.874022 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.873981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e878d8b-f462-4730-8dea-d80a8315a227-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj7bk\" (UID: \"3e878d8b-f462-4730-8dea-d80a8315a227\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"
Sep 11 21:56:02.874022 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.874028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trg8t\" (UniqueName: \"kubernetes.io/projected/3e878d8b-f462-4730-8dea-d80a8315a227-kube-api-access-trg8t\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj7bk\" (UID: \"3e878d8b-f462-4730-8dea-d80a8315a227\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"
Sep 11 21:56:02.975216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.975177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e878d8b-f462-4730-8dea-d80a8315a227-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj7bk\" (UID: \"3e878d8b-f462-4730-8dea-d80a8315a227\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"
Sep 11 21:56:02.975380 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.975232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trg8t\" (UniqueName: \"kubernetes.io/projected/3e878d8b-f462-4730-8dea-d80a8315a227-kube-api-access-trg8t\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj7bk\" (UID: \"3e878d8b-f462-4730-8dea-d80a8315a227\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"
Sep 11 21:56:02.975677 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.975629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e878d8b-f462-4730-8dea-d80a8315a227-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj7bk\" (UID: \"3e878d8b-f462-4730-8dea-d80a8315a227\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"
Sep 11 21:56:02.984489 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:02.984466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trg8t\" (UniqueName: \"kubernetes.io/projected/3e878d8b-f462-4730-8dea-d80a8315a227-kube-api-access-trg8t\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj7bk\" (UID: \"3e878d8b-f462-4730-8dea-d80a8315a227\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"
Sep 11 21:56:03.103497 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:03.103394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"
Sep 11 21:56:03.383561 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:03.383478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk"]
Sep 11 21:56:03.387163 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:56:03.387130    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e878d8b_f462_4730_8dea_d80a8315a227.slice/crio-232958162a1f9be1df1681d691c40a1db421128e787529634bd667b6fb15778f WatchSource:0}: Error finding container 232958162a1f9be1df1681d691c40a1db421128e787529634bd667b6fb15778f: Status 404 returned error can't find the container with id 232958162a1f9be1df1681d691c40a1db421128e787529634bd667b6fb15778f
Sep 11 21:56:03.976911 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:03.976877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk" event={"ID":"3e878d8b-f462-4730-8dea-d80a8315a227","Type":"ContainerStarted","Data":"232958162a1f9be1df1681d691c40a1db421128e787529634bd667b6fb15778f"}
Sep 11 21:56:05.985952 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:05.985912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk" event={"ID":"3e878d8b-f462-4730-8dea-d80a8315a227","Type":"ContainerStarted","Data":"271836cc6895c8d309b96b2b13bf61ecffaa5ae99978f1a939ca75317c086aab"}
Sep 11 21:56:06.007012 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:06.006939    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj7bk" podStartSLOduration=2.287366928 podStartE2EDuration="4.006923724s" totalImagesPullingTime="1.719556796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:06.005696111 +0000 UTC m=+227.690642822" watchObservedRunningTime="2026-09-11 21:56:06.006923724 +0000 UTC m=+227.691870438"
Sep 11 21:56:07.493811 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.493776    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"]
Sep 11 21:56:07.497120 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.497102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.499303 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.499283    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 21:56:07.499768 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.499751    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 21:56:07.499878 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.499782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9nhgg\""
Sep 11 21:56:07.504875 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.504848    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"]
Sep 11 21:56:07.510984 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.510957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.511111 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.510993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztnq\" (UniqueName: \"kubernetes.io/projected/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-kube-api-access-dztnq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.511111 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.511016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.611874 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.611831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.611874 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.611881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztnq\" (UniqueName: \"kubernetes.io/projected/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-kube-api-access-dztnq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.612129 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.611996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.612249 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.612227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.612297 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.612268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.620233 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.620206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztnq\" (UniqueName: \"kubernetes.io/projected/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-kube-api-access-dztnq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:07.807383 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:07.807293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:08.089321 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:08.089237    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"]
Sep 11 21:56:08.092536 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:56:08.092509    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aa7fa2_d437_4da6_b963_7bbfe1187ebc.slice/crio-c5c56fca9c2738eae459861f7ddbaa9bf542668d6744dd5656413262c1fbaa45 WatchSource:0}: Error finding container c5c56fca9c2738eae459861f7ddbaa9bf542668d6744dd5656413262c1fbaa45: Status 404 returned error can't find the container with id c5c56fca9c2738eae459861f7ddbaa9bf542668d6744dd5656413262c1fbaa45
Sep 11 21:56:08.998107 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:08.998071    2578 generic.go:356] "Generic (PLEG): container finished" podID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerID="73641e5c4c6be229eb6c2e9d2d598935e85924e6a6b29b0d6d9ac6f8913f1977" exitCode=0
Sep 11 21:56:08.998602 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:08.998124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v" event={"ID":"51aa7fa2-d437-4da6-b963-7bbfe1187ebc","Type":"ContainerDied","Data":"73641e5c4c6be229eb6c2e9d2d598935e85924e6a6b29b0d6d9ac6f8913f1977"}
Sep 11 21:56:08.998602 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:08.998165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v" event={"ID":"51aa7fa2-d437-4da6-b963-7bbfe1187ebc","Type":"ContainerStarted","Data":"c5c56fca9c2738eae459861f7ddbaa9bf542668d6744dd5656413262c1fbaa45"}
Sep 11 21:56:14.021212 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:14.021138    2578 generic.go:356] "Generic (PLEG): container finished" podID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerID="1ff82043b2ff23f4d68c810558109b53b922caf71ffa1bf3f7f0fd44eaf6d1bc" exitCode=0
Sep 11 21:56:14.021746 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:14.021221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v" event={"ID":"51aa7fa2-d437-4da6-b963-7bbfe1187ebc","Type":"ContainerDied","Data":"1ff82043b2ff23f4d68c810558109b53b922caf71ffa1bf3f7f0fd44eaf6d1bc"}
Sep 11 21:56:15.027689 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:15.027656    2578 generic.go:356] "Generic (PLEG): container finished" podID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerID="109f727d413f9a9ca8f7a7760444ba3e701ae1251c0209c3d2bae90bd545b830" exitCode=0
Sep 11 21:56:15.028100 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:15.027729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v" event={"ID":"51aa7fa2-d437-4da6-b963-7bbfe1187ebc","Type":"ContainerDied","Data":"109f727d413f9a9ca8f7a7760444ba3e701ae1251c0209c3d2bae90bd545b830"}
Sep 11 21:56:16.156696 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.156670    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:16.170062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.170033    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ll59r"]
Sep 11 21:56:16.170284 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.170273    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerName="pull"
Sep 11 21:56:16.170322 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.170285    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerName="pull"
Sep 11 21:56:16.170322 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.170300    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerName="extract"
Sep 11 21:56:16.170322 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.170305    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerName="extract"
Sep 11 21:56:16.170322 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.170317    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerName="util"
Sep 11 21:56:16.170322 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.170322    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerName="util"
Sep 11 21:56:16.170474 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.170364    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="51aa7fa2-d437-4da6-b963-7bbfe1187ebc" containerName="extract"
Sep 11 21:56:16.173709 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.173693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ll59r"
Sep 11 21:56:16.175877 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.175852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 21:56:16.175997 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.175966    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 21:56:16.176139 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.176121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dztnq\" (UniqueName: \"kubernetes.io/projected/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-kube-api-access-dztnq\") pod \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") "
Sep 11 21:56:16.176202 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.176165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-util\") pod \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") "
Sep 11 21:56:16.176202 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.176185    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-48mkc\""
Sep 11 21:56:16.176286 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.176269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-bundle\") pod \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\" (UID: \"51aa7fa2-d437-4da6-b963-7bbfe1187ebc\") "
Sep 11 21:56:16.176623 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.176594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-bundle" (OuterVolumeSpecName: "bundle") pod "51aa7fa2-d437-4da6-b963-7bbfe1187ebc" (UID: "51aa7fa2-d437-4da6-b963-7bbfe1187ebc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:16.186180 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.185891    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-util" (OuterVolumeSpecName: "util") pod "51aa7fa2-d437-4da6-b963-7bbfe1187ebc" (UID: "51aa7fa2-d437-4da6-b963-7bbfe1187ebc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:16.186288 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.186255    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-kube-api-access-dztnq" (OuterVolumeSpecName: "kube-api-access-dztnq") pod "51aa7fa2-d437-4da6-b963-7bbfe1187ebc" (UID: "51aa7fa2-d437-4da6-b963-7bbfe1187ebc"). InnerVolumeSpecName "kube-api-access-dztnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:16.190665 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.190622    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ll59r"]
Sep 11 21:56:16.277537 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.277499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdgb2\" (UniqueName: \"kubernetes.io/projected/e972c151-1e90-4ddb-bb14-c95294a646f9-kube-api-access-pdgb2\") pod \"cert-manager-55b4d954d5-ll59r\" (UID: \"e972c151-1e90-4ddb-bb14-c95294a646f9\") " pod="cert-manager/cert-manager-55b4d954d5-ll59r"
Sep 11 21:56:16.277537 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.277538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e972c151-1e90-4ddb-bb14-c95294a646f9-bound-sa-token\") pod \"cert-manager-55b4d954d5-ll59r\" (UID: \"e972c151-1e90-4ddb-bb14-c95294a646f9\") " pod="cert-manager/cert-manager-55b4d954d5-ll59r"
Sep 11 21:56:16.277763 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.277619    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dztnq\" (UniqueName: \"kubernetes.io/projected/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-kube-api-access-dztnq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:16.277763 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.277662    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-util\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:16.277763 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.277675    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa7fa2-d437-4da6-b963-7bbfe1187ebc-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:16.378811 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.378665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdgb2\" (UniqueName: \"kubernetes.io/projected/e972c151-1e90-4ddb-bb14-c95294a646f9-kube-api-access-pdgb2\") pod \"cert-manager-55b4d954d5-ll59r\" (UID: \"e972c151-1e90-4ddb-bb14-c95294a646f9\") " pod="cert-manager/cert-manager-55b4d954d5-ll59r"
Sep 11 21:56:16.378811 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.378805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e972c151-1e90-4ddb-bb14-c95294a646f9-bound-sa-token\") pod \"cert-manager-55b4d954d5-ll59r\" (UID: \"e972c151-1e90-4ddb-bb14-c95294a646f9\") " pod="cert-manager/cert-manager-55b4d954d5-ll59r"
Sep 11 21:56:16.386372 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.386346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e972c151-1e90-4ddb-bb14-c95294a646f9-bound-sa-token\") pod \"cert-manager-55b4d954d5-ll59r\" (UID: \"e972c151-1e90-4ddb-bb14-c95294a646f9\") " pod="cert-manager/cert-manager-55b4d954d5-ll59r"
Sep 11 21:56:16.386525 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.386506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdgb2\" (UniqueName: \"kubernetes.io/projected/e972c151-1e90-4ddb-bb14-c95294a646f9-kube-api-access-pdgb2\") pod \"cert-manager-55b4d954d5-ll59r\" (UID: \"e972c151-1e90-4ddb-bb14-c95294a646f9\") " pod="cert-manager/cert-manager-55b4d954d5-ll59r"
Sep 11 21:56:16.491658 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.491600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ll59r"
Sep 11 21:56:16.761818 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:16.761786    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ll59r"]
Sep 11 21:56:16.765527 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:56:16.765494    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode972c151_1e90_4ddb_bb14_c95294a646f9.slice/crio-c6a215b325e6a41c4f738f2f5ae00f4e4baa0842f31d42ca9ef086714d276e7d WatchSource:0}: Error finding container c6a215b325e6a41c4f738f2f5ae00f4e4baa0842f31d42ca9ef086714d276e7d: Status 404 returned error can't find the container with id c6a215b325e6a41c4f738f2f5ae00f4e4baa0842f31d42ca9ef086714d276e7d
Sep 11 21:56:17.037314 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:17.037228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v" event={"ID":"51aa7fa2-d437-4da6-b963-7bbfe1187ebc","Type":"ContainerDied","Data":"c5c56fca9c2738eae459861f7ddbaa9bf542668d6744dd5656413262c1fbaa45"}
Sep 11 21:56:17.037314 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:17.037244    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft7c6v"
Sep 11 21:56:17.037314 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:17.037279    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c56fca9c2738eae459861f7ddbaa9bf542668d6744dd5656413262c1fbaa45"
Sep 11 21:56:17.038584 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:17.038561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ll59r" event={"ID":"e972c151-1e90-4ddb-bb14-c95294a646f9","Type":"ContainerStarted","Data":"c6a215b325e6a41c4f738f2f5ae00f4e4baa0842f31d42ca9ef086714d276e7d"}
Sep 11 21:56:20.191344 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.191306    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"]
Sep 11 21:56:20.196136 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.196116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"
Sep 11 21:56:20.198222 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.198201    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 21:56:20.198723 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.198700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jrkpc\""
Sep 11 21:56:20.198723 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.198712    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:56:20.202745 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.202717    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"]
Sep 11 21:56:20.309675 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.309604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6479942c-c6af-4cb5-8ff9-39c867d76981-tmp\") pod \"openshift-lws-operator-fd8ccff4c-827v4\" (UID: \"6479942c-c6af-4cb5-8ff9-39c867d76981\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"
Sep 11 21:56:20.309675 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.309667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5q7\" (UniqueName: \"kubernetes.io/projected/6479942c-c6af-4cb5-8ff9-39c867d76981-kube-api-access-dh5q7\") pod \"openshift-lws-operator-fd8ccff4c-827v4\" (UID: \"6479942c-c6af-4cb5-8ff9-39c867d76981\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"
Sep 11 21:56:20.410831 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.410720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6479942c-c6af-4cb5-8ff9-39c867d76981-tmp\") pod \"openshift-lws-operator-fd8ccff4c-827v4\" (UID: \"6479942c-c6af-4cb5-8ff9-39c867d76981\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"
Sep 11 21:56:20.410831 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.410766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5q7\" (UniqueName: \"kubernetes.io/projected/6479942c-c6af-4cb5-8ff9-39c867d76981-kube-api-access-dh5q7\") pod \"openshift-lws-operator-fd8ccff4c-827v4\" (UID: \"6479942c-c6af-4cb5-8ff9-39c867d76981\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"
Sep 11 21:56:20.411178 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.411153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6479942c-c6af-4cb5-8ff9-39c867d76981-tmp\") pod \"openshift-lws-operator-fd8ccff4c-827v4\" (UID: \"6479942c-c6af-4cb5-8ff9-39c867d76981\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"
Sep 11 21:56:20.420225 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.420196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5q7\" (UniqueName: \"kubernetes.io/projected/6479942c-c6af-4cb5-8ff9-39c867d76981-kube-api-access-dh5q7\") pod \"openshift-lws-operator-fd8ccff4c-827v4\" (UID: \"6479942c-c6af-4cb5-8ff9-39c867d76981\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"
Sep 11 21:56:20.505561 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.505485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"
Sep 11 21:56:20.777938 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:20.777860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4"]
Sep 11 21:56:20.781030 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:56:20.781002    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6479942c_c6af_4cb5_8ff9_39c867d76981.slice/crio-024b26de8b956f08a93efff2aac77783d9bad502eb2a917361b7a707c1b5e95f WatchSource:0}: Error finding container 024b26de8b956f08a93efff2aac77783d9bad502eb2a917361b7a707c1b5e95f: Status 404 returned error can't find the container with id 024b26de8b956f08a93efff2aac77783d9bad502eb2a917361b7a707c1b5e95f
Sep 11 21:56:21.055922 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:21.055843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4" event={"ID":"6479942c-c6af-4cb5-8ff9-39c867d76981","Type":"ContainerStarted","Data":"024b26de8b956f08a93efff2aac77783d9bad502eb2a917361b7a707c1b5e95f"}
Sep 11 21:56:24.072134 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:24.072092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ll59r" event={"ID":"e972c151-1e90-4ddb-bb14-c95294a646f9","Type":"ContainerStarted","Data":"15927c31af24ef2be990693124f1140a5d1253bc4047cff59aa3843424632fed"}
Sep 11 21:56:24.073853 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:24.073825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4" event={"ID":"6479942c-c6af-4cb5-8ff9-39c867d76981","Type":"ContainerStarted","Data":"5309a25845530a6f7a2a39e6985341bc9a4698d9cab173c818c65b279caa60a4"}
Sep 11 21:56:24.090845 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:24.090779    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ll59r" podStartSLOduration=1.518466751 podStartE2EDuration="8.090762041s" totalImagesPullingTime="6.57229529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:24.090312469 +0000 UTC m=+245.775259182" watchObservedRunningTime="2026-09-11 21:56:24.090762041 +0000 UTC m=+245.775708754"
Sep 11 21:56:24.108420 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:24.108359    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-827v4" podStartSLOduration=1.550706393 podStartE2EDuration="4.108340445s" totalImagesPullingTime="2.557634052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:24.106417169 +0000 UTC m=+245.791363885" watchObservedRunningTime="2026-09-11 21:56:24.108340445 +0000 UTC m=+245.793287159"
Sep 11 21:56:26.551790 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.551752    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"]
Sep 11 21:56:26.600496 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.600456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"]
Sep 11 21:56:26.600685 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.600582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.602921 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.602893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 21:56:26.603422 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.603404    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 21:56:26.603491 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.603404    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9nhgg\""
Sep 11 21:56:26.663515 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.663470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.663723 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.663522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kjv\" (UniqueName: \"kubernetes.io/projected/7f9b6f09-e114-4429-bee7-32aad17dca27-kube-api-access-m6kjv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.663723 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.663651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.764370 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.764338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.764560 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.764393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.764560 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.764427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kjv\" (UniqueName: \"kubernetes.io/projected/7f9b6f09-e114-4429-bee7-32aad17dca27-kube-api-access-m6kjv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.764819 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.764795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.764890 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.764830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.772491 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.772465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kjv\" (UniqueName: \"kubernetes.io/projected/7f9b6f09-e114-4429-bee7-32aad17dca27-kube-api-access-m6kjv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:26.910431 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:26.910348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:27.197698 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:27.197666    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"]
Sep 11 21:56:27.201152 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:56:27.201118    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9b6f09_e114_4429_bee7_32aad17dca27.slice/crio-2d92a9f6dd1380619bc46231d8f6064d5845f694a2edff6c557ddb88c1f1ae09 WatchSource:0}: Error finding container 2d92a9f6dd1380619bc46231d8f6064d5845f694a2edff6c557ddb88c1f1ae09: Status 404 returned error can't find the container with id 2d92a9f6dd1380619bc46231d8f6064d5845f694a2edff6c557ddb88c1f1ae09
Sep 11 21:56:28.092036 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:28.091951    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerID="ea22fa4fe22be656337eddc0d75463789b4fed2b65f7d71c7d6626fad4ee5fbf" exitCode=0
Sep 11 21:56:28.092036 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:28.092008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87" event={"ID":"7f9b6f09-e114-4429-bee7-32aad17dca27","Type":"ContainerDied","Data":"ea22fa4fe22be656337eddc0d75463789b4fed2b65f7d71c7d6626fad4ee5fbf"}
Sep 11 21:56:28.092428 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:28.092042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87" event={"ID":"7f9b6f09-e114-4429-bee7-32aad17dca27","Type":"ContainerStarted","Data":"2d92a9f6dd1380619bc46231d8f6064d5845f694a2edff6c557ddb88c1f1ae09"}
Sep 11 21:56:29.099008 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:29.098914    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerID="7c42b956692def66a37fffda432c9e0aa800208507faff89b8025fb870115b95" exitCode=0
Sep 11 21:56:29.099349 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:29.099006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87" event={"ID":"7f9b6f09-e114-4429-bee7-32aad17dca27","Type":"ContainerDied","Data":"7c42b956692def66a37fffda432c9e0aa800208507faff89b8025fb870115b95"}
Sep 11 21:56:30.105709 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:30.105665    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerID="ba3c85a4a2d73f2cf92124faa057db235b42006e66b0a52ce6df1e147490ad70" exitCode=0
Sep 11 21:56:30.106090 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:30.105739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87" event={"ID":"7f9b6f09-e114-4429-bee7-32aad17dca27","Type":"ContainerDied","Data":"ba3c85a4a2d73f2cf92124faa057db235b42006e66b0a52ce6df1e147490ad70"}
Sep 11 21:56:31.230480 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.230448    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:31.300876 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.300836    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-bundle\") pod \"7f9b6f09-e114-4429-bee7-32aad17dca27\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") "
Sep 11 21:56:31.301055 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.300893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6kjv\" (UniqueName: \"kubernetes.io/projected/7f9b6f09-e114-4429-bee7-32aad17dca27-kube-api-access-m6kjv\") pod \"7f9b6f09-e114-4429-bee7-32aad17dca27\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") "
Sep 11 21:56:31.301055 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.300947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-util\") pod \"7f9b6f09-e114-4429-bee7-32aad17dca27\" (UID: \"7f9b6f09-e114-4429-bee7-32aad17dca27\") "
Sep 11 21:56:31.301689 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.301655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-bundle" (OuterVolumeSpecName: "bundle") pod "7f9b6f09-e114-4429-bee7-32aad17dca27" (UID: "7f9b6f09-e114-4429-bee7-32aad17dca27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:31.303106 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.303082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9b6f09-e114-4429-bee7-32aad17dca27-kube-api-access-m6kjv" (OuterVolumeSpecName: "kube-api-access-m6kjv") pod "7f9b6f09-e114-4429-bee7-32aad17dca27" (UID: "7f9b6f09-e114-4429-bee7-32aad17dca27"). InnerVolumeSpecName "kube-api-access-m6kjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:31.306872 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.306848    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-util" (OuterVolumeSpecName: "util") pod "7f9b6f09-e114-4429-bee7-32aad17dca27" (UID: "7f9b6f09-e114-4429-bee7-32aad17dca27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:31.402418 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.402309    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6kjv\" (UniqueName: \"kubernetes.io/projected/7f9b6f09-e114-4429-bee7-32aad17dca27-kube-api-access-m6kjv\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:31.402418 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.402356    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-util\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:31.402418 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:31.402366    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9b6f09-e114-4429-bee7-32aad17dca27-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:32.116255 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:32.116229    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87"
Sep 11 21:56:32.116420 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:32.116227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9m87" event={"ID":"7f9b6f09-e114-4429-bee7-32aad17dca27","Type":"ContainerDied","Data":"2d92a9f6dd1380619bc46231d8f6064d5845f694a2edff6c557ddb88c1f1ae09"}
Sep 11 21:56:32.116420 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:32.116353    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d92a9f6dd1380619bc46231d8f6064d5845f694a2edff6c557ddb88c1f1ae09"
Sep 11 21:56:43.739470 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.739389    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"]
Sep 11 21:56:43.740070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.739701    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerName="extract"
Sep 11 21:56:43.740070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.739716    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerName="extract"
Sep 11 21:56:43.740070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.739730    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerName="pull"
Sep 11 21:56:43.740070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.739736    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerName="pull"
Sep 11 21:56:43.740070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.739747    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerName="util"
Sep 11 21:56:43.740070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.739755    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerName="util"
Sep 11 21:56:43.740070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.739803    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f9b6f09-e114-4429-bee7-32aad17dca27" containerName="extract"
Sep 11 21:56:43.742792 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.742771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.746260 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.746237    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 21:56:43.746410 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.746272    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 21:56:43.746410 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.746308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bfn6m\""
Sep 11 21:56:43.746410 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.746337    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 21:56:43.746778 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.746759    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 21:56:43.758647 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.758612    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"]
Sep 11 21:56:43.893214 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.893176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmm7\" (UniqueName: \"kubernetes.io/projected/0ad12098-b953-49ec-8fbe-9907801c0c62-kube-api-access-smmm7\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.893383 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.893229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ad12098-b953-49ec-8fbe-9907801c0c62-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.893383 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.893299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad12098-b953-49ec-8fbe-9907801c0c62-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.893383 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.893361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ad12098-b953-49ec-8fbe-9907801c0c62-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.893494 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.893386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ad12098-b953-49ec-8fbe-9907801c0c62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.994320 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.994235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ad12098-b953-49ec-8fbe-9907801c0c62-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.994320 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.994281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad12098-b953-49ec-8fbe-9907801c0c62-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.994320 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.994313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ad12098-b953-49ec-8fbe-9907801c0c62-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.994694 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.994338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ad12098-b953-49ec-8fbe-9907801c0c62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.994694 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.994575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmm7\" (UniqueName: \"kubernetes.io/projected/0ad12098-b953-49ec-8fbe-9907801c0c62-kube-api-access-smmm7\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.994816 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.994751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ad12098-b953-49ec-8fbe-9907801c0c62-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.996755 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.996724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad12098-b953-49ec-8fbe-9907801c0c62-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.997066 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.997048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ad12098-b953-49ec-8fbe-9907801c0c62-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:43.997132 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:43.997064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ad12098-b953-49ec-8fbe-9907801c0c62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:44.029626 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.029591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmm7\" (UniqueName: \"kubernetes.io/projected/0ad12098-b953-49ec-8fbe-9907801c0c62-kube-api-access-smmm7\") pod \"opendatahub-operator-controller-manager-5bf8845876-jpctb\" (UID: \"0ad12098-b953-49ec-8fbe-9907801c0c62\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:44.052553 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.052517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:56:44.091333 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.091298    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"]
Sep 11 21:56:44.094743 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.094720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.100170 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.099791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 21:56:44.100258 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.100175    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9nhgg\""
Sep 11 21:56:44.106159 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.106130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 21:56:44.114184 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.114150    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"]
Sep 11 21:56:44.195762 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.195725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lcq\" (UniqueName: \"kubernetes.io/projected/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-kube-api-access-w7lcq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.195946 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.195844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.195946 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.195876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.297087 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.296993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.297087 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.297029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.297087 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.297060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lcq\" (UniqueName: \"kubernetes.io/projected/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-kube-api-access-w7lcq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.297395 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.297373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.297462 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.297444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.309124 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.309097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lcq\" (UniqueName: \"kubernetes.io/projected/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-kube-api-access-w7lcq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.328608 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.328577    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"]
Sep 11 21:56:44.331786 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:56:44.331761    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad12098_b953_49ec_8fbe_9907801c0c62.slice/crio-31d04a83b3b757312b6e69b86a03401141d22de309da2d718f1cb2990d8563ea WatchSource:0}: Error finding container 31d04a83b3b757312b6e69b86a03401141d22de309da2d718f1cb2990d8563ea: Status 404 returned error can't find the container with id 31d04a83b3b757312b6e69b86a03401141d22de309da2d718f1cb2990d8563ea
Sep 11 21:56:44.407413 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.407375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:44.685760 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:44.685726    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"]
Sep 11 21:56:44.688644 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:56:44.688605    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f31e81_3fe7_43a2_8fe8_c0ce0035011d.slice/crio-80870af30dfde7a77cb157a091f4c9735608d5f1ff0c1a719849d4c31690b7da WatchSource:0}: Error finding container 80870af30dfde7a77cb157a091f4c9735608d5f1ff0c1a719849d4c31690b7da: Status 404 returned error can't find the container with id 80870af30dfde7a77cb157a091f4c9735608d5f1ff0c1a719849d4c31690b7da
Sep 11 21:56:45.173523 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:45.173334    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerID="94b46fd97562af42443a1950fc769903c6f769f15bdbeffd737bf4b946ae776b" exitCode=0
Sep 11 21:56:45.173523 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:45.173427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg" event={"ID":"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d","Type":"ContainerDied","Data":"94b46fd97562af42443a1950fc769903c6f769f15bdbeffd737bf4b946ae776b"}
Sep 11 21:56:45.173523 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:45.173480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg" event={"ID":"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d","Type":"ContainerStarted","Data":"80870af30dfde7a77cb157a091f4c9735608d5f1ff0c1a719849d4c31690b7da"}
Sep 11 21:56:45.175542 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:45.175501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb" event={"ID":"0ad12098-b953-49ec-8fbe-9907801c0c62","Type":"ContainerStarted","Data":"31d04a83b3b757312b6e69b86a03401141d22de309da2d718f1cb2990d8563ea"}
Sep 11 21:56:48.192839 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:48.192796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb" event={"ID":"0ad12098-b953-49ec-8fbe-9907801c0c62","Type":"ContainerStarted","Data":"9c94ee3d86457ec34fa3224819984028825674a5eba7f876eb7843c1f4525ac4"}
Sep 11 21:56:48.194629 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:48.194601    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerID="7ce9bc312b79980eec66641ad2ff247b52f0cca4b7aebd29cbf25e56a10b6e6c" exitCode=0
Sep 11 21:56:48.194792 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:48.194671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg" event={"ID":"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d","Type":"ContainerDied","Data":"7ce9bc312b79980eec66641ad2ff247b52f0cca4b7aebd29cbf25e56a10b6e6c"}
Sep 11 21:56:49.200413 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:49.200384    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ad12098-b953-49ec-8fbe-9907801c0c62" containerID="9c94ee3d86457ec34fa3224819984028825674a5eba7f876eb7843c1f4525ac4" exitCode=0
Sep 11 21:56:49.200911 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:49.200484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb" event={"ID":"0ad12098-b953-49ec-8fbe-9907801c0c62","Type":"ContainerDied","Data":"9c94ee3d86457ec34fa3224819984028825674a5eba7f876eb7843c1f4525ac4"}
Sep 11 21:56:49.202824 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:49.202800    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerID="1cdf547049c6f5718d8750013e9af627ace11b445d1ad1c374c937f82950dc43" exitCode=0
Sep 11 21:56:49.202937 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:49.202846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg" event={"ID":"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d","Type":"ContainerDied","Data":"1cdf547049c6f5718d8750013e9af627ace11b445d1ad1c374c937f82950dc43"}
Sep 11 21:56:50.209026 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.208986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb" event={"ID":"0ad12098-b953-49ec-8fbe-9907801c0c62","Type":"ContainerStarted","Data":"e23e4e93e6016f5507b6a31e83c0d93d13fa11413c4b83adabcc658d382dd578"}
Sep 11 21:56:50.231462 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.231407    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb" podStartSLOduration=2.775948745 podStartE2EDuration="7.231391907s" totalImagesPullingTime="3.280647851s" totalInitContainerRuntime="1.174795311s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:50.229554891 +0000 UTC m=+271.914501615" watchObservedRunningTime="2026-09-11 21:56:50.231391907 +0000 UTC m=+271.916338621"
Sep 11 21:56:50.339404 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.339376    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:50.446575 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.446541    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-util\") pod \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") "
Sep 11 21:56:50.446575 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.446585    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7lcq\" (UniqueName: \"kubernetes.io/projected/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-kube-api-access-w7lcq\") pod \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") "
Sep 11 21:56:50.446788 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.446617    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-bundle\") pod \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\" (UID: \"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d\") "
Sep 11 21:56:50.447321 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.447293    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-bundle" (OuterVolumeSpecName: "bundle") pod "b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" (UID: "b0f31e81-3fe7-43a2-8fe8-c0ce0035011d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:50.448546 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.448521    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-kube-api-access-w7lcq" (OuterVolumeSpecName: "kube-api-access-w7lcq") pod "b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" (UID: "b0f31e81-3fe7-43a2-8fe8-c0ce0035011d"). InnerVolumeSpecName "kube-api-access-w7lcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:50.450995 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.450952    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-util" (OuterVolumeSpecName: "util") pod "b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" (UID: "b0f31e81-3fe7-43a2-8fe8-c0ce0035011d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:50.547751 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.547667    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-util\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:50.547751 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.547696    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7lcq\" (UniqueName: \"kubernetes.io/projected/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-kube-api-access-w7lcq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:50.547751 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:50.547707    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0f31e81-3fe7-43a2-8fe8-c0ce0035011d-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:56:51.215345 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:51.215312    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg"
Sep 11 21:56:51.215345 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:51.215331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbwgg" event={"ID":"b0f31e81-3fe7-43a2-8fe8-c0ce0035011d","Type":"ContainerDied","Data":"80870af30dfde7a77cb157a091f4c9735608d5f1ff0c1a719849d4c31690b7da"}
Sep 11 21:56:51.215806 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:51.215373    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80870af30dfde7a77cb157a091f4c9735608d5f1ff0c1a719849d4c31690b7da"
Sep 11 21:56:51.215806 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:56:51.215751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:57:02.223165 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:02.223134    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-jpctb"
Sep 11 21:57:04.758798 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.758762    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"]
Sep 11 21:57:04.759174 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.759035    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerName="util"
Sep 11 21:57:04.759174 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.759046    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerName="util"
Sep 11 21:57:04.759174 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.759056    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerName="extract"
Sep 11 21:57:04.759174 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.759062    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerName="extract"
Sep 11 21:57:04.759174 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.759081    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerName="pull"
Sep 11 21:57:04.759174 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.759087    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerName="pull"
Sep 11 21:57:04.759174 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.759131    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f31e81-3fe7-43a2-8fe8-c0ce0035011d" containerName="extract"
Sep 11 21:57:04.763473 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.763456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:04.766156 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.766134    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 21:57:04.766396 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.766379    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 21:57:04.766396 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.766389    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 21:57:04.766547 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.766428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 21:57:04.787871 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.787847    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"]
Sep 11 21:57:04.858592 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.858562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/524ca142-8f37-48be-b646-5ac01dc75c1c-tmp\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:04.858794 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.858610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlk7\" (UniqueName: \"kubernetes.io/projected/524ca142-8f37-48be-b646-5ac01dc75c1c-kube-api-access-mdlk7\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:04.858794 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.858663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524ca142-8f37-48be-b646-5ac01dc75c1c-tls-certs\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:04.959393 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.959352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/524ca142-8f37-48be-b646-5ac01dc75c1c-tmp\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:04.959571 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.959409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlk7\" (UniqueName: \"kubernetes.io/projected/524ca142-8f37-48be-b646-5ac01dc75c1c-kube-api-access-mdlk7\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:04.959571 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.959437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524ca142-8f37-48be-b646-5ac01dc75c1c-tls-certs\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:04.959571 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:57:04.959535    2578 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 21:57:04.959722 ip-10-0-141-45 kubenswrapper[2578]: E0911 21:57:04.959596    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524ca142-8f37-48be-b646-5ac01dc75c1c-tls-certs podName:524ca142-8f37-48be-b646-5ac01dc75c1c nodeName:}" failed. No retries permitted until 2026-09-11 21:57:05.459581049 +0000 UTC m=+287.144527742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/524ca142-8f37-48be-b646-5ac01dc75c1c-tls-certs") pod "kube-auth-proxy-576b4b4bb7-8g9g5" (UID: "524ca142-8f37-48be-b646-5ac01dc75c1c") : secret "kube-auth-proxy-tls" not found
Sep 11 21:57:04.961755 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.961724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/524ca142-8f37-48be-b646-5ac01dc75c1c-tmp\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:04.975854 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:04.975830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlk7\" (UniqueName: \"kubernetes.io/projected/524ca142-8f37-48be-b646-5ac01dc75c1c-kube-api-access-mdlk7\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:05.463977 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:05.463943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524ca142-8f37-48be-b646-5ac01dc75c1c-tls-certs\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:05.466332 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:05.466308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524ca142-8f37-48be-b646-5ac01dc75c1c-tls-certs\") pod \"kube-auth-proxy-576b4b4bb7-8g9g5\" (UID: \"524ca142-8f37-48be-b646-5ac01dc75c1c\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:05.672968 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:05.672934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"
Sep 11 21:57:05.943835 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:05.943806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5"]
Sep 11 21:57:05.946319 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:57:05.946294    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524ca142_8f37_48be_b646_5ac01dc75c1c.slice/crio-26ab4b9022d179eb3ebc186c45a1fea163df2054069589d85d1984cf38fb3b0e WatchSource:0}: Error finding container 26ab4b9022d179eb3ebc186c45a1fea163df2054069589d85d1984cf38fb3b0e: Status 404 returned error can't find the container with id 26ab4b9022d179eb3ebc186c45a1fea163df2054069589d85d1984cf38fb3b0e
Sep 11 21:57:06.284937 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:06.284903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5" event={"ID":"524ca142-8f37-48be-b646-5ac01dc75c1c","Type":"ContainerStarted","Data":"26ab4b9022d179eb3ebc186c45a1fea163df2054069589d85d1984cf38fb3b0e"}
Sep 11 21:57:09.303704 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:09.303670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5" event={"ID":"524ca142-8f37-48be-b646-5ac01dc75c1c","Type":"ContainerStarted","Data":"4431e8a498f70cb1ba0dcfe224d928b0736ccb84a54d367d32dd97160ecc7e33"}
Sep 11 21:57:09.320420 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:09.320370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-8g9g5" podStartSLOduration=2.446893483 podStartE2EDuration="5.320354239s" totalImagesPullingTime="2.873460756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:09.319324565 +0000 UTC m=+291.004271272" watchObservedRunningTime="2026-09-11 21:57:09.320354239 +0000 UTC m=+291.005300953"
Sep 11 21:57:12.428554 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.428519    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"]
Sep 11 21:57:12.431846 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.431823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.434162 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.434136    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:57:12.434287 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.434192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rd2h6\""
Sep 11 21:57:12.444490 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.444466    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"]
Sep 11 21:57:12.523531 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93eece28-e770-46f9-890e-4b59e58cbacc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.523531 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7lr\" (UniqueName: \"kubernetes.io/projected/93eece28-e770-46f9-890e-4b59e58cbacc-kube-api-access-dk7lr\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.523772 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.523772 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.523772 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93eece28-e770-46f9-890e-4b59e58cbacc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.523772 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93eece28-e770-46f9-890e-4b59e58cbacc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.523772 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.524002 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.524002 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.523807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624237 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93eece28-e770-46f9-890e-4b59e58cbacc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624237 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93eece28-e770-46f9-890e-4b59e58cbacc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624447 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624447 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624552 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624552 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93eece28-e770-46f9-890e-4b59e58cbacc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624670 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7lr\" (UniqueName: \"kubernetes.io/projected/93eece28-e770-46f9-890e-4b59e58cbacc-kube-api-access-dk7lr\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624737 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624798 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.624876 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.624849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.625094 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.625073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.625177 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.625074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93eece28-e770-46f9-890e-4b59e58cbacc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.625177 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.625134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.625325 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.625264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.626825 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.626795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93eece28-e770-46f9-890e-4b59e58cbacc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.627034 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.627014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93eece28-e770-46f9-890e-4b59e58cbacc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.632561 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.632536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93eece28-e770-46f9-890e-4b59e58cbacc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.632827 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.632807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7lr\" (UniqueName: \"kubernetes.io/projected/93eece28-e770-46f9-890e-4b59e58cbacc-kube-api-access-dk7lr\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx\" (UID: \"93eece28-e770-46f9-890e-4b59e58cbacc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:12.741398 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:12.741317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:13.015310 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:13.015232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"]
Sep 11 21:57:13.018116 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:57:13.018089    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93eece28_e770_46f9_890e_4b59e58cbacc.slice/crio-8137cded7cc446bf0ae59a9aa0d280f2aa903ed19605dbfe1c2b7ea04f8ba4e3 WatchSource:0}: Error finding container 8137cded7cc446bf0ae59a9aa0d280f2aa903ed19605dbfe1c2b7ea04f8ba4e3: Status 404 returned error can't find the container with id 8137cded7cc446bf0ae59a9aa0d280f2aa903ed19605dbfe1c2b7ea04f8ba4e3
Sep 11 21:57:13.322106 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:13.322025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx" event={"ID":"93eece28-e770-46f9-890e-4b59e58cbacc","Type":"ContainerStarted","Data":"8137cded7cc446bf0ae59a9aa0d280f2aa903ed19605dbfe1c2b7ea04f8ba4e3"}
Sep 11 21:57:18.744380 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:18.744351    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 21:57:21.986976 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:21.986934    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"]
Sep 11 21:57:21.990493 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:21.990474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:21.992878 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:21.992851    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 21:57:21.992878 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:21.992867    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z26f5\""
Sep 11 21:57:21.993062 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:21.992974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 21:57:21.993334 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:21.993318    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 21:57:22.005339 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.005308    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"]
Sep 11 21:57:22.103305 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.103269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6sn\" (UniqueName: \"kubernetes.io/projected/6b0db6eb-463f-4c90-90b6-00ce94afd71b-kube-api-access-tc6sn\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.103479 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.103317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b0db6eb-463f-4c90-90b6-00ce94afd71b-manager-config\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.103479 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.103338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0db6eb-463f-4c90-90b6-00ce94afd71b-cert\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.103479 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.103440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b0db6eb-463f-4c90-90b6-00ce94afd71b-metrics-cert\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.204525 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.204486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6sn\" (UniqueName: \"kubernetes.io/projected/6b0db6eb-463f-4c90-90b6-00ce94afd71b-kube-api-access-tc6sn\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.204735 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.204550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b0db6eb-463f-4c90-90b6-00ce94afd71b-manager-config\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.204735 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.204575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0db6eb-463f-4c90-90b6-00ce94afd71b-cert\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.204735 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.204625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b0db6eb-463f-4c90-90b6-00ce94afd71b-metrics-cert\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.205365 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.205335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b0db6eb-463f-4c90-90b6-00ce94afd71b-manager-config\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.207276 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.207248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0db6eb-463f-4c90-90b6-00ce94afd71b-cert\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.207421 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.207370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b0db6eb-463f-4c90-90b6-00ce94afd71b-metrics-cert\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.217556 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.217519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6sn\" (UniqueName: \"kubernetes.io/projected/6b0db6eb-463f-4c90-90b6-00ce94afd71b-kube-api-access-tc6sn\") pod \"lws-controller-manager-57494c65d5-t5768\" (UID: \"6b0db6eb-463f-4c90-90b6-00ce94afd71b\") " pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.299372 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.299277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:22.581457 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.581323    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"]
Sep 11 21:57:22.583879 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:57:22.583854    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0db6eb_463f_4c90_90b6_00ce94afd71b.slice/crio-bba182f48886ea49fb02993225136bf0555fabfded35ea24280bfaddda73a411 WatchSource:0}: Error finding container bba182f48886ea49fb02993225136bf0555fabfded35ea24280bfaddda73a411: Status 404 returned error can't find the container with id bba182f48886ea49fb02993225136bf0555fabfded35ea24280bfaddda73a411
Sep 11 21:57:22.585844 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:22.585827    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:57:23.373890 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:23.373850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768" event={"ID":"6b0db6eb-463f-4c90-90b6-00ce94afd71b","Type":"ContainerStarted","Data":"bba182f48886ea49fb02993225136bf0555fabfded35ea24280bfaddda73a411"}
Sep 11 21:57:24.814183 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:24.814129    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 21:57:24.814571 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:24.814234    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 21:57:24.814571 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:24.814284    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 21:57:25.385974 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:25.385934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx" event={"ID":"93eece28-e770-46f9-890e-4b59e58cbacc","Type":"ContainerStarted","Data":"acf120e5217e1b11c0856a1153e9db11952b8c6061cfaa9e5e8ea4e7a10a63fb"}
Sep 11 21:57:25.387557 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:25.387529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768" event={"ID":"6b0db6eb-463f-4c90-90b6-00ce94afd71b","Type":"ContainerStarted","Data":"91992365642f98d1a74eabd58536c8dc97c55f7e4ce0cdc99e4f6bbfdacb35ef"}
Sep 11 21:57:25.387694 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:25.387619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:57:25.441872 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:25.441811    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx" podStartSLOduration=1.648190775 podStartE2EDuration="13.441792062s" totalImagesPullingTime="11.793601287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:25.439625075 +0000 UTC m=+307.124571801" watchObservedRunningTime="2026-09-11 21:57:25.441792062 +0000 UTC m=+307.126738777"
Sep 11 21:57:25.540402 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:25.540337    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768" podStartSLOduration=2.312789115 podStartE2EDuration="4.540317619s" totalImagesPullingTime="2.227528504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:25.539127177 +0000 UTC m=+307.224073904" watchObservedRunningTime="2026-09-11 21:57:25.540317619 +0000 UTC m=+307.225264333"
Sep 11 21:57:25.741697 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:25.741660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:25.746032 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:25.746002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:26.391928 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:26.391893    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:26.392876 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:26.392857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx"
Sep 11 21:57:36.393688 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:57:36.393648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57494c65d5-t5768"
Sep 11 21:58:20.199363 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.199279    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"]
Sep 11 21:58:20.202569 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.202551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.204744 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.204717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 21:58:20.204901 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.204832    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9nhgg\""
Sep 11 21:58:20.204901 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.204873    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 21:58:20.211314 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.211293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"]
Sep 11 21:58:20.288948 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.288901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgwv\" (UniqueName: \"kubernetes.io/projected/96be68b0-6283-4c9c-a551-4981fcd97daa-kube-api-access-4dgwv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.289139 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.288976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.289139 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.289100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.297803 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.297775    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"]
Sep 11 21:58:20.301120 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.301102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.309411 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.309390    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"]
Sep 11 21:58:20.390159 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.390121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.390347 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.390204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgwv\" (UniqueName: \"kubernetes.io/projected/96be68b0-6283-4c9c-a551-4981fcd97daa-kube-api-access-4dgwv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.390347 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.390235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfzh\" (UniqueName: \"kubernetes.io/projected/992d0384-ae19-4bb3-bead-9a1fab2348db-kube-api-access-4qfzh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.390347 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.390263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.390347 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.390298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.390347 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.390321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.390556 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.390460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.390677 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.390653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.398603 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.398577    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"]
Sep 11 21:58:20.400070 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.400046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgwv\" (UniqueName: \"kubernetes.io/projected/96be68b0-6283-4c9c-a551-4981fcd97daa-kube-api-access-4dgwv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.407011 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.406994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.410954 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.410922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"]
Sep 11 21:58:20.491052 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.490965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfzh\" (UniqueName: \"kubernetes.io/projected/992d0384-ae19-4bb3-bead-9a1fab2348db-kube-api-access-4qfzh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.491052 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.491003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72xms\" (UniqueName: \"kubernetes.io/projected/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-kube-api-access-72xms\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.491052 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.491023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.491052 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.491042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.491298 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.491173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.491298 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.491230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.491500 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.491485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.491535 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.491517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.501464 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.501437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfzh\" (UniqueName: \"kubernetes.io/projected/992d0384-ae19-4bb3-bead-9a1fab2348db-kube-api-access-4qfzh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.511842 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.511815    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"]
Sep 11 21:58:20.512951 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.512933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:20.516009 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.515994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.525139 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.525046    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"]
Sep 11 21:58:20.592422 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.592392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72xms\" (UniqueName: \"kubernetes.io/projected/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-kube-api-access-72xms\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.592422 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.592425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.592677 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.592461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.592677 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.592502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.592677 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.592529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4l5k\" (UniqueName: \"kubernetes.io/projected/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-kube-api-access-l4l5k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.592677 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.592565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.592900 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.592822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.592994 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.592968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.600427 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.600402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72xms\" (UniqueName: \"kubernetes.io/projected/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-kube-api-access-72xms\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.611217 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.611194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:20.693481 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.693433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.693481 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.693483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4l5k\" (UniqueName: \"kubernetes.io/projected/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-kube-api-access-l4l5k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.693691 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.693582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.693944 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.693926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.693980 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.693924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.701382 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.701362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4l5k\" (UniqueName: \"kubernetes.io/projected/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-kube-api-access-l4l5k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.717137 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.717112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:20.789434 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.789367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"]
Sep 11 21:58:20.792067 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:58:20.792038    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96be68b0_6283_4c9c_a551_4981fcd97daa.slice/crio-b8570bbf040c0d98da97d3c07a3539b7fd6af6e99eae71fc1dcd96a873c042de WatchSource:0}: Error finding container b8570bbf040c0d98da97d3c07a3539b7fd6af6e99eae71fc1dcd96a873c042de: Status 404 returned error can't find the container with id b8570bbf040c0d98da97d3c07a3539b7fd6af6e99eae71fc1dcd96a873c042de
Sep 11 21:58:20.840676 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.840654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:20.880838 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.880800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"]
Sep 11 21:58:20.884880 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:58:20.884847    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992d0384_ae19_4bb3_bead_9a1fab2348db.slice/crio-8549354ce640a39537c640e3df22ee8128e76f0a8bbd87782bb63017f55cc8bf WatchSource:0}: Error finding container 8549354ce640a39537c640e3df22ee8128e76f0a8bbd87782bb63017f55cc8bf: Status 404 returned error can't find the container with id 8549354ce640a39537c640e3df22ee8128e76f0a8bbd87782bb63017f55cc8bf
Sep 11 21:58:20.985840 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:20.985736    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"]
Sep 11 21:58:20.987996 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:58:20.987968    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d5ffc4_20aa_440d_ba71_3b79b2714b4f.slice/crio-570be63584ad50893ed297ec1ba16668335083a3c9e11618b6e61781250c9df8 WatchSource:0}: Error finding container 570be63584ad50893ed297ec1ba16668335083a3c9e11618b6e61781250c9df8: Status 404 returned error can't find the container with id 570be63584ad50893ed297ec1ba16668335083a3c9e11618b6e61781250c9df8
Sep 11 21:58:21.122941 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.122915    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"]
Sep 11 21:58:21.145446 ip-10-0-141-45 kubenswrapper[2578]: W0911 21:58:21.145412    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d94aa3_36ab_4057_ae31_7f8e7eb8f76d.slice/crio-3cff4c0c45fa60e582f92b20969c6ddb983f4462402331e3ce68350ea5193dc2 WatchSource:0}: Error finding container 3cff4c0c45fa60e582f92b20969c6ddb983f4462402331e3ce68350ea5193dc2: Status 404 returned error can't find the container with id 3cff4c0c45fa60e582f92b20969c6ddb983f4462402331e3ce68350ea5193dc2
Sep 11 21:58:21.646449 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.646415    2578 generic.go:356] "Generic (PLEG): container finished" podID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerID="28aabe9dba3e732c9aa5444ad731af2feab100d99b3584ee705a6c16324a469d" exitCode=0
Sep 11 21:58:21.646891 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.646503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg" event={"ID":"98d5ffc4-20aa-440d-ba71-3b79b2714b4f","Type":"ContainerDied","Data":"28aabe9dba3e732c9aa5444ad731af2feab100d99b3584ee705a6c16324a469d"}
Sep 11 21:58:21.646891 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.646564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg" event={"ID":"98d5ffc4-20aa-440d-ba71-3b79b2714b4f","Type":"ContainerStarted","Data":"570be63584ad50893ed297ec1ba16668335083a3c9e11618b6e61781250c9df8"}
Sep 11 21:58:21.648104 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.648086    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerID="1aed26ff1d609bd48d6eb42ebe809599f99c5c78a0a9283afbc870456217cc3e" exitCode=0
Sep 11 21:58:21.648188 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.648167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn" event={"ID":"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d","Type":"ContainerDied","Data":"1aed26ff1d609bd48d6eb42ebe809599f99c5c78a0a9283afbc870456217cc3e"}
Sep 11 21:58:21.648232 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.648217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn" event={"ID":"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d","Type":"ContainerStarted","Data":"3cff4c0c45fa60e582f92b20969c6ddb983f4462402331e3ce68350ea5193dc2"}
Sep 11 21:58:21.649793 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.649769    2578 generic.go:356] "Generic (PLEG): container finished" podID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerID="a30320aed5c2e0938dfebcfad4c5e05adeeafc78daf7fdf36b404a7f43607285" exitCode=0
Sep 11 21:58:21.649874 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.649849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8" event={"ID":"992d0384-ae19-4bb3-bead-9a1fab2348db","Type":"ContainerDied","Data":"a30320aed5c2e0938dfebcfad4c5e05adeeafc78daf7fdf36b404a7f43607285"}
Sep 11 21:58:21.649934 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.649892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8" event={"ID":"992d0384-ae19-4bb3-bead-9a1fab2348db","Type":"ContainerStarted","Data":"8549354ce640a39537c640e3df22ee8128e76f0a8bbd87782bb63017f55cc8bf"}
Sep 11 21:58:21.651597 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.651577    2578 generic.go:356] "Generic (PLEG): container finished" podID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerID="ce57d6b4103f749f566ca113315ab14a9cc9d2ed13eb20dce2dbfa3a9f19fca1" exitCode=0
Sep 11 21:58:21.651674 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.651614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p" event={"ID":"96be68b0-6283-4c9c-a551-4981fcd97daa","Type":"ContainerDied","Data":"ce57d6b4103f749f566ca113315ab14a9cc9d2ed13eb20dce2dbfa3a9f19fca1"}
Sep 11 21:58:21.651674 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:21.651665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p" event={"ID":"96be68b0-6283-4c9c-a551-4981fcd97daa","Type":"ContainerStarted","Data":"b8570bbf040c0d98da97d3c07a3539b7fd6af6e99eae71fc1dcd96a873c042de"}
Sep 11 21:58:22.659575 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:22.659542    2578 generic.go:356] "Generic (PLEG): container finished" podID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerID="63d666d502c1d8d635fa1bc8eda4038715fcdebcff8f42fee14faeb7d76ea3b7" exitCode=0
Sep 11 21:58:22.659963 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:22.659621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8" event={"ID":"992d0384-ae19-4bb3-bead-9a1fab2348db","Type":"ContainerDied","Data":"63d666d502c1d8d635fa1bc8eda4038715fcdebcff8f42fee14faeb7d76ea3b7"}
Sep 11 21:58:23.666601 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:23.666565    2578 generic.go:356] "Generic (PLEG): container finished" podID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerID="01c90501c2b3e198ee19c1157d47f360c2b92898c9e599412d5a5ab6abeac0bb" exitCode=0
Sep 11 21:58:23.667035 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:23.666667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg" event={"ID":"98d5ffc4-20aa-440d-ba71-3b79b2714b4f","Type":"ContainerDied","Data":"01c90501c2b3e198ee19c1157d47f360c2b92898c9e599412d5a5ab6abeac0bb"}
Sep 11 21:58:23.669102 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:23.669081    2578 generic.go:356] "Generic (PLEG): container finished" podID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerID="3680be95251e0dc5a96276706880d9292a9422894937c687bf0be8f23f00991f" exitCode=0
Sep 11 21:58:23.669209 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:23.669170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8" event={"ID":"992d0384-ae19-4bb3-bead-9a1fab2348db","Type":"ContainerDied","Data":"3680be95251e0dc5a96276706880d9292a9422894937c687bf0be8f23f00991f"}
Sep 11 21:58:23.671005 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:23.670981    2578 generic.go:356] "Generic (PLEG): container finished" podID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerID="e166914b18c53a78b852d4fc259cd24d00627beccc28b6a49872a8a2d2710f14" exitCode=0
Sep 11 21:58:23.671113 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:23.671036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p" event={"ID":"96be68b0-6283-4c9c-a551-4981fcd97daa","Type":"ContainerDied","Data":"e166914b18c53a78b852d4fc259cd24d00627beccc28b6a49872a8a2d2710f14"}
Sep 11 21:58:24.678491 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.678450    2578 generic.go:356] "Generic (PLEG): container finished" podID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerID="5de34ed0bf051d7f38adaf0165669cf4da8b9e9de195d6b2b23ed42487f904e0" exitCode=0
Sep 11 21:58:24.678929 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.678532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg" event={"ID":"98d5ffc4-20aa-440d-ba71-3b79b2714b4f","Type":"ContainerDied","Data":"5de34ed0bf051d7f38adaf0165669cf4da8b9e9de195d6b2b23ed42487f904e0"}
Sep 11 21:58:24.680700 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.680676    2578 generic.go:356] "Generic (PLEG): container finished" podID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerID="ee5c47d92eefaf8d5e5bbc2a7cbebcd8716d5dccdfa3ce52e442ba005ca10f42" exitCode=0
Sep 11 21:58:24.680800 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.680756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p" event={"ID":"96be68b0-6283-4c9c-a551-4981fcd97daa","Type":"ContainerDied","Data":"ee5c47d92eefaf8d5e5bbc2a7cbebcd8716d5dccdfa3ce52e442ba005ca10f42"}
Sep 11 21:58:24.803216 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.803190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:24.831351 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.831322    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-util\") pod \"992d0384-ae19-4bb3-bead-9a1fab2348db\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") "
Sep 11 21:58:24.831476 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.831375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-bundle\") pod \"992d0384-ae19-4bb3-bead-9a1fab2348db\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") "
Sep 11 21:58:24.831476 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.831458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qfzh\" (UniqueName: \"kubernetes.io/projected/992d0384-ae19-4bb3-bead-9a1fab2348db-kube-api-access-4qfzh\") pod \"992d0384-ae19-4bb3-bead-9a1fab2348db\" (UID: \"992d0384-ae19-4bb3-bead-9a1fab2348db\") "
Sep 11 21:58:24.831832 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.831807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-bundle" (OuterVolumeSpecName: "bundle") pod "992d0384-ae19-4bb3-bead-9a1fab2348db" (UID: "992d0384-ae19-4bb3-bead-9a1fab2348db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:24.833483 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.833460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992d0384-ae19-4bb3-bead-9a1fab2348db-kube-api-access-4qfzh" (OuterVolumeSpecName: "kube-api-access-4qfzh") pod "992d0384-ae19-4bb3-bead-9a1fab2348db" (UID: "992d0384-ae19-4bb3-bead-9a1fab2348db"). InnerVolumeSpecName "kube-api-access-4qfzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:58:24.840372 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.840346    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-util" (OuterVolumeSpecName: "util") pod "992d0384-ae19-4bb3-bead-9a1fab2348db" (UID: "992d0384-ae19-4bb3-bead-9a1fab2348db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:24.932690 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.932579    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qfzh\" (UniqueName: \"kubernetes.io/projected/992d0384-ae19-4bb3-bead-9a1fab2348db-kube-api-access-4qfzh\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:24.932690 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.932616    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-util\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:24.932690 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:24.932627    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992d0384-ae19-4bb3-bead-9a1fab2348db-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:25.687294 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.687265    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8"
Sep 11 21:58:25.687294 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.687280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x6wj8" event={"ID":"992d0384-ae19-4bb3-bead-9a1fab2348db","Type":"ContainerDied","Data":"8549354ce640a39537c640e3df22ee8128e76f0a8bbd87782bb63017f55cc8bf"}
Sep 11 21:58:25.687804 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.687327    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8549354ce640a39537c640e3df22ee8128e76f0a8bbd87782bb63017f55cc8bf"
Sep 11 21:58:25.818571 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.818547    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:25.838124 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.838097    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:25.941690 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.941570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-bundle\") pod \"96be68b0-6283-4c9c-a551-4981fcd97daa\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") "
Sep 11 21:58:25.941690 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.941667    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-bundle\") pod \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") "
Sep 11 21:58:25.941899 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.941726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-util\") pod \"96be68b0-6283-4c9c-a551-4981fcd97daa\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") "
Sep 11 21:58:25.941899 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.941749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72xms\" (UniqueName: \"kubernetes.io/projected/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-kube-api-access-72xms\") pod \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") "
Sep 11 21:58:25.941899 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.941786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dgwv\" (UniqueName: \"kubernetes.io/projected/96be68b0-6283-4c9c-a551-4981fcd97daa-kube-api-access-4dgwv\") pod \"96be68b0-6283-4c9c-a551-4981fcd97daa\" (UID: \"96be68b0-6283-4c9c-a551-4981fcd97daa\") "
Sep 11 21:58:25.941899 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.941819    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-util\") pod \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\" (UID: \"98d5ffc4-20aa-440d-ba71-3b79b2714b4f\") "
Sep 11 21:58:25.942708 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.942628    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-bundle" (OuterVolumeSpecName: "bundle") pod "96be68b0-6283-4c9c-a551-4981fcd97daa" (UID: "96be68b0-6283-4c9c-a551-4981fcd97daa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:25.942708 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.942679    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-bundle" (OuterVolumeSpecName: "bundle") pod "98d5ffc4-20aa-440d-ba71-3b79b2714b4f" (UID: "98d5ffc4-20aa-440d-ba71-3b79b2714b4f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:25.944205 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.944175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-kube-api-access-72xms" (OuterVolumeSpecName: "kube-api-access-72xms") pod "98d5ffc4-20aa-440d-ba71-3b79b2714b4f" (UID: "98d5ffc4-20aa-440d-ba71-3b79b2714b4f"). InnerVolumeSpecName "kube-api-access-72xms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:58:25.944415 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.944390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96be68b0-6283-4c9c-a551-4981fcd97daa-kube-api-access-4dgwv" (OuterVolumeSpecName: "kube-api-access-4dgwv") pod "96be68b0-6283-4c9c-a551-4981fcd97daa" (UID: "96be68b0-6283-4c9c-a551-4981fcd97daa"). InnerVolumeSpecName "kube-api-access-4dgwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:58:25.948101 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.948073    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-util" (OuterVolumeSpecName: "util") pod "98d5ffc4-20aa-440d-ba71-3b79b2714b4f" (UID: "98d5ffc4-20aa-440d-ba71-3b79b2714b4f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:25.948389 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:25.948369    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-util" (OuterVolumeSpecName: "util") pod "96be68b0-6283-4c9c-a551-4981fcd97daa" (UID: "96be68b0-6283-4c9c-a551-4981fcd97daa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:26.043294 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.043255    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-util\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:26.043294 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.043287    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72xms\" (UniqueName: \"kubernetes.io/projected/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-kube-api-access-72xms\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:26.043294 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.043299    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dgwv\" (UniqueName: \"kubernetes.io/projected/96be68b0-6283-4c9c-a551-4981fcd97daa-kube-api-access-4dgwv\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:26.043518 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.043308    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-util\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:26.043518 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.043316    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96be68b0-6283-4c9c-a551-4981fcd97daa-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:26.043518 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.043324    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98d5ffc4-20aa-440d-ba71-3b79b2714b4f-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:26.694872 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.694840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p"
Sep 11 21:58:26.694872 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.694859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a89b4p" event={"ID":"96be68b0-6283-4c9c-a551-4981fcd97daa","Type":"ContainerDied","Data":"b8570bbf040c0d98da97d3c07a3539b7fd6af6e99eae71fc1dcd96a873c042de"}
Sep 11 21:58:26.695378 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.694911    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8570bbf040c0d98da97d3c07a3539b7fd6af6e99eae71fc1dcd96a873c042de"
Sep 11 21:58:26.697008 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.696984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg" event={"ID":"98d5ffc4-20aa-440d-ba71-3b79b2714b4f","Type":"ContainerDied","Data":"570be63584ad50893ed297ec1ba16668335083a3c9e11618b6e61781250c9df8"}
Sep 11 21:58:26.697105 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.697020    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570be63584ad50893ed297ec1ba16668335083a3c9e11618b6e61781250c9df8"
Sep 11 21:58:26.697105 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:26.696991    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvcwg"
Sep 11 21:58:40.769240 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:40.769151    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerID="e15c8e61fb5ae6722c5f854b091cbf6772cc8035f4d974e6da10028b51ec5b38" exitCode=0
Sep 11 21:58:40.769605 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:40.769238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn" event={"ID":"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d","Type":"ContainerDied","Data":"e15c8e61fb5ae6722c5f854b091cbf6772cc8035f4d974e6da10028b51ec5b38"}
Sep 11 21:58:41.775976 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:41.775941    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerID="e82ac71a4c4d73a837baae396ad135d28e4bbbdfd27d4a7980c5a7476b60e5d1" exitCode=0
Sep 11 21:58:41.776355 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:41.776031    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn" event={"ID":"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d","Type":"ContainerDied","Data":"e82ac71a4c4d73a837baae396ad135d28e4bbbdfd27d4a7980c5a7476b60e5d1"}
Sep 11 21:58:42.901879 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:42.901850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:42.990220 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:42.990185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-bundle\") pod \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") "
Sep 11 21:58:42.990220 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:42.990225    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4l5k\" (UniqueName: \"kubernetes.io/projected/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-kube-api-access-l4l5k\") pod \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") "
Sep 11 21:58:42.990444 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:42.990345    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-util\") pod \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\" (UID: \"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d\") "
Sep 11 21:58:42.990747 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:42.990725    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-bundle" (OuterVolumeSpecName: "bundle") pod "f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" (UID: "f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:42.992431 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:42.992405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-kube-api-access-l4l5k" (OuterVolumeSpecName: "kube-api-access-l4l5k") pod "f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" (UID: "f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d"). InnerVolumeSpecName "kube-api-access-l4l5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:58:42.996589 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:42.996564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-util" (OuterVolumeSpecName: "util") pod "f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" (UID: "f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:43.091615 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:43.091535    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-util\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:43.091615 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:43.091565    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:43.091615 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:43.091574    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4l5k\" (UniqueName: \"kubernetes.io/projected/f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d-kube-api-access-l4l5k\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 21:58:43.787382 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:43.787355    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn"
Sep 11 21:58:43.787556 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:43.787385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p8khn" event={"ID":"f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d","Type":"ContainerDied","Data":"3cff4c0c45fa60e582f92b20969c6ddb983f4462402331e3ce68350ea5193dc2"}
Sep 11 21:58:43.787556 ip-10-0-141-45 kubenswrapper[2578]: I0911 21:58:43.787435    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cff4c0c45fa60e582f92b20969c6ddb983f4462402331e3ce68350ea5193dc2"
Sep 11 22:00:00.321624 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.321585    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwmpb"]
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322016    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerName="pull"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322037    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerName="pull"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322053    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerName="extract"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322061    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerName="extract"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322072    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerName="extract"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322080    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerName="extract"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322097    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerName="pull"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322105    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerName="pull"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322118    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerName="util"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322125    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerName="util"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322136    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerName="extract"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322144    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerName="extract"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322159    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerName="util"
Sep 11 22:00:00.322165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322166    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerName="util"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322177    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerName="pull"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322185    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerName="pull"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322192    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerName="pull"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322200    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerName="pull"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322210    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerName="util"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322218    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerName="util"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322228    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerName="extract"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322236    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerName="extract"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322248    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerName="util"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322256    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerName="util"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322329    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="992d0384-ae19-4bb3-bead-9a1fab2348db" containerName="extract"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322342    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d5ffc4-20aa-440d-ba71-3b79b2714b4f" containerName="extract"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322354    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d94aa3-36ab-4057-ae31-7f8e7eb8f76d" containerName="extract"
Sep 11 22:00:00.322856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.322364    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="96be68b0-6283-4c9c-a551-4981fcd97daa" containerName="extract"
Sep 11 22:00:00.325600 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.325578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb"
Sep 11 22:00:00.327725 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.327707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 22:00:00.327839 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.327759    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-74fxz\""
Sep 11 22:00:00.327839 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.327786    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 22:00:00.331867 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.331844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwmpb"]
Sep 11 22:00:00.346542 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.346513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7bjb\" (UniqueName: \"kubernetes.io/projected/e90a3a10-8aeb-44dc-a5af-dd121f7a36bf-kube-api-access-t7bjb\") pod \"authorino-7bcfd8f9cd-gwmpb\" (UID: \"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb"
Sep 11 22:00:00.447068 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.447029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7bjb\" (UniqueName: \"kubernetes.io/projected/e90a3a10-8aeb-44dc-a5af-dd121f7a36bf-kube-api-access-t7bjb\") pod \"authorino-7bcfd8f9cd-gwmpb\" (UID: \"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb"
Sep 11 22:00:00.460518 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.460480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7bjb\" (UniqueName: \"kubernetes.io/projected/e90a3a10-8aeb-44dc-a5af-dd121f7a36bf-kube-api-access-t7bjb\") pod \"authorino-7bcfd8f9cd-gwmpb\" (UID: \"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb"
Sep 11 22:00:00.540558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.540516    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-k7sms"]
Sep 11 22:00:00.543986 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.543966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k7sms"
Sep 11 22:00:00.548453 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.548422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvgh\" (UniqueName: \"kubernetes.io/projected/3a295379-e7f9-46ad-ab9d-5fdad67cc847-kube-api-access-gbvgh\") pod \"authorino-59c974bd48-k7sms\" (UID: \"3a295379-e7f9-46ad-ab9d-5fdad67cc847\") " pod="kuadrant-system/authorino-59c974bd48-k7sms"
Sep 11 22:00:00.550919 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.550894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k7sms"]
Sep 11 22:00:00.635065 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.634981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb"
Sep 11 22:00:00.649456 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.649427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvgh\" (UniqueName: \"kubernetes.io/projected/3a295379-e7f9-46ad-ab9d-5fdad67cc847-kube-api-access-gbvgh\") pod \"authorino-59c974bd48-k7sms\" (UID: \"3a295379-e7f9-46ad-ab9d-5fdad67cc847\") " pod="kuadrant-system/authorino-59c974bd48-k7sms"
Sep 11 22:00:00.657626 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.657596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvgh\" (UniqueName: \"kubernetes.io/projected/3a295379-e7f9-46ad-ab9d-5fdad67cc847-kube-api-access-gbvgh\") pod \"authorino-59c974bd48-k7sms\" (UID: \"3a295379-e7f9-46ad-ab9d-5fdad67cc847\") " pod="kuadrant-system/authorino-59c974bd48-k7sms"
Sep 11 22:00:00.854008 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.853978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k7sms"
Sep 11 22:00:00.909241 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:00.909209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwmpb"]
Sep 11 22:00:01.127985 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:01.127953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k7sms"]
Sep 11 22:00:01.130250 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:00:01.130222    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a295379_e7f9_46ad_ab9d_5fdad67cc847.slice/crio-0868939bc622435ac31c1492a038de8d9c732e42b9dee4d376fea68e091c3916 WatchSource:0}: Error finding container 0868939bc622435ac31c1492a038de8d9c732e42b9dee4d376fea68e091c3916: Status 404 returned error can't find the container with id 0868939bc622435ac31c1492a038de8d9c732e42b9dee4d376fea68e091c3916
Sep 11 22:00:01.147064 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:01.147028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k7sms" event={"ID":"3a295379-e7f9-46ad-ab9d-5fdad67cc847","Type":"ContainerStarted","Data":"0868939bc622435ac31c1492a038de8d9c732e42b9dee4d376fea68e091c3916"}
Sep 11 22:00:01.148175 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:01.148145    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb" event={"ID":"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf","Type":"ContainerStarted","Data":"2996853ada45f08ad946f9a877ea90281e5ddf63a91d833fc9cf591e36664fbe"}
Sep 11 22:00:05.173781 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:05.173744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k7sms" event={"ID":"3a295379-e7f9-46ad-ab9d-5fdad67cc847","Type":"ContainerStarted","Data":"b55a5d05ca326eeaf14b09bcd5a9bb10f3fac17092c3339452972415d6505cd8"}
Sep 11 22:00:05.175251 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:05.175228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb" event={"ID":"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf","Type":"ContainerStarted","Data":"7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04"}
Sep 11 22:00:05.189714 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:05.189653    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-k7sms" podStartSLOduration=1.847941346 podStartE2EDuration="5.189620356s" totalImagesPullingTime="3.34167901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:05.187475849 +0000 UTC m=+466.872422575" watchObservedRunningTime="2026-09-11 22:00:05.189620356 +0000 UTC m=+466.874567069"
Sep 11 22:00:05.200993 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:05.200931    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb" podStartSLOduration=1.653310177 podStartE2EDuration="5.200908459s" totalImagesPullingTime="3.547598282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:05.200015086 +0000 UTC m=+466.884961800" watchObservedRunningTime="2026-09-11 22:00:05.200908459 +0000 UTC m=+466.885855173"
Sep 11 22:00:05.227385 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:05.227349    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwmpb"]
Sep 11 22:00:07.185857 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:07.185787    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb" podUID="e90a3a10-8aeb-44dc-a5af-dd121f7a36bf" containerName="authorino" containerID="cri-o://7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04" gracePeriod=30
Sep 11 22:00:07.436937 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:07.436869    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb"
Sep 11 22:00:07.494224 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:07.494183    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7bjb\" (UniqueName: \"kubernetes.io/projected/e90a3a10-8aeb-44dc-a5af-dd121f7a36bf-kube-api-access-t7bjb\") pod \"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf\" (UID: \"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf\") "
Sep 11 22:00:07.496395 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:07.496369    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90a3a10-8aeb-44dc-a5af-dd121f7a36bf-kube-api-access-t7bjb" (OuterVolumeSpecName: "kube-api-access-t7bjb") pod "e90a3a10-8aeb-44dc-a5af-dd121f7a36bf" (UID: "e90a3a10-8aeb-44dc-a5af-dd121f7a36bf"). InnerVolumeSpecName "kube-api-access-t7bjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:00:07.594922 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:07.594885    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7bjb\" (UniqueName: \"kubernetes.io/projected/e90a3a10-8aeb-44dc-a5af-dd121f7a36bf-kube-api-access-t7bjb\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:00:08.192363 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.192327    2578 generic.go:356] "Generic (PLEG): container finished" podID="e90a3a10-8aeb-44dc-a5af-dd121f7a36bf" containerID="7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04" exitCode=0
Sep 11 22:00:08.192852 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.192372    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb"
Sep 11 22:00:08.192852 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.192416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb" event={"ID":"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf","Type":"ContainerDied","Data":"7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04"}
Sep 11 22:00:08.192852 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.192464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gwmpb" event={"ID":"e90a3a10-8aeb-44dc-a5af-dd121f7a36bf","Type":"ContainerDied","Data":"2996853ada45f08ad946f9a877ea90281e5ddf63a91d833fc9cf591e36664fbe"}
Sep 11 22:00:08.192852 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.192489    2578 scope.go:117] "RemoveContainer" containerID="7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04"
Sep 11 22:00:08.201335 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.201317    2578 scope.go:117] "RemoveContainer" containerID="7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04"
Sep 11 22:00:08.201609 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:00:08.201592    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04\": container with ID starting with 7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04 not found: ID does not exist" containerID="7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04"
Sep 11 22:00:08.201742 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.201623    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04"} err="failed to get container status \"7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04\": rpc error: code = NotFound desc = could not find container \"7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04\": container with ID starting with 7cd807d1838979acdc4d9a96dc455f2efc6bfd3b513fa8af4afc5f2c98b65e04 not found: ID does not exist"
Sep 11 22:00:08.212229 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.212207    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwmpb"]
Sep 11 22:00:08.215609 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.215586    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwmpb"]
Sep 11 22:00:08.856607 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:08.856569    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90a3a10-8aeb-44dc-a5af-dd121f7a36bf" path="/var/lib/kubelet/pods/e90a3a10-8aeb-44dc-a5af-dd121f7a36bf/volumes"
Sep 11 22:00:16.168254 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.168214    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"]
Sep 11 22:00:16.168740 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.168552    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90a3a10-8aeb-44dc-a5af-dd121f7a36bf" containerName="authorino"
Sep 11 22:00:16.168740 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.168565    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90a3a10-8aeb-44dc-a5af-dd121f7a36bf" containerName="authorino"
Sep 11 22:00:16.168740 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.168624    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90a3a10-8aeb-44dc-a5af-dd121f7a36bf" containerName="authorino"
Sep 11 22:00:16.171391 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.171369    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:16.173504 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.173478    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 22:00:16.173628 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.173500    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:00:16.174165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.174131    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:00:16.174267 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.174187    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lhwxk\""
Sep 11 22:00:16.180822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.180798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"]
Sep 11 22:00:16.258399 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.258355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfh2\" (UniqueName: \"kubernetes.io/projected/0a43fa42-09c2-457b-b1ac-ce4860f754a5-kube-api-access-zwfh2\") pod \"postgres-69fb4594cf-sp24z\" (UID: \"0a43fa42-09c2-457b-b1ac-ce4860f754a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:16.258597 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.258411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a43fa42-09c2-457b-b1ac-ce4860f754a5-data\") pod \"postgres-69fb4594cf-sp24z\" (UID: \"0a43fa42-09c2-457b-b1ac-ce4860f754a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:16.359480 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.359425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfh2\" (UniqueName: \"kubernetes.io/projected/0a43fa42-09c2-457b-b1ac-ce4860f754a5-kube-api-access-zwfh2\") pod \"postgres-69fb4594cf-sp24z\" (UID: \"0a43fa42-09c2-457b-b1ac-ce4860f754a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:16.359742 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.359513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a43fa42-09c2-457b-b1ac-ce4860f754a5-data\") pod \"postgres-69fb4594cf-sp24z\" (UID: \"0a43fa42-09c2-457b-b1ac-ce4860f754a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:16.359953 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.359931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a43fa42-09c2-457b-b1ac-ce4860f754a5-data\") pod \"postgres-69fb4594cf-sp24z\" (UID: \"0a43fa42-09c2-457b-b1ac-ce4860f754a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:16.367626 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.367596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfh2\" (UniqueName: \"kubernetes.io/projected/0a43fa42-09c2-457b-b1ac-ce4860f754a5-kube-api-access-zwfh2\") pod \"postgres-69fb4594cf-sp24z\" (UID: \"0a43fa42-09c2-457b-b1ac-ce4860f754a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:16.482525 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.482427    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:16.753859 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:16.753770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"]
Sep 11 22:00:16.757128 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:00:16.757098    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a43fa42_09c2_457b_b1ac_ce4860f754a5.slice/crio-a97e4af75afcdca781615cbb6b458e8f805bc5ef8440094fd079a05b087cfdcc WatchSource:0}: Error finding container a97e4af75afcdca781615cbb6b458e8f805bc5ef8440094fd079a05b087cfdcc: Status 404 returned error can't find the container with id a97e4af75afcdca781615cbb6b458e8f805bc5ef8440094fd079a05b087cfdcc
Sep 11 22:00:17.242305 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:17.242272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z" event={"ID":"0a43fa42-09c2-457b-b1ac-ce4860f754a5","Type":"ContainerStarted","Data":"a97e4af75afcdca781615cbb6b458e8f805bc5ef8440094fd079a05b087cfdcc"}
Sep 11 22:00:23.477223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:23.477198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 22:00:24.289448 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:24.289413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z" event={"ID":"0a43fa42-09c2-457b-b1ac-ce4860f754a5","Type":"ContainerStarted","Data":"f86315a8cb537f9e26dc8dccfeea413d695dc86350f18345ea0e0156d4a32699"}
Sep 11 22:00:24.289621 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:24.289571    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:24.306188 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:24.306138    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z" podStartSLOduration=1.589774464 podStartE2EDuration="8.306123797s" totalImagesPullingTime="6.716349333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:24.303764388 +0000 UTC m=+485.988711103" watchObservedRunningTime="2026-09-11 22:00:24.306123797 +0000 UTC m=+485.991070547"
Sep 11 22:00:30.323212 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:30.323178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sp24z"
Sep 11 22:00:30.867198 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:30.867163    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dw5nh"]
Sep 11 22:00:30.873461 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:30.873438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dw5nh"
Sep 11 22:00:30.876471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:30.876443    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dw5nh"]
Sep 11 22:00:30.987160 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:30.987121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7p2m\" (UniqueName: \"kubernetes.io/projected/3426216f-e850-4777-a1af-9fa6624d3648-kube-api-access-m7p2m\") pod \"authorino-557f876f69-dw5nh\" (UID: \"3426216f-e850-4777-a1af-9fa6624d3648\") " pod="kuadrant-system/authorino-557f876f69-dw5nh"
Sep 11 22:00:31.003108 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.003067    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dw5nh"]
Sep 11 22:00:31.003330 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:00:31.003311    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-m7p2m], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-dw5nh" podUID="3426216f-e850-4777-a1af-9fa6624d3648"
Sep 11 22:00:31.027203 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.027161    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xnl6p"]
Sep 11 22:00:31.031125 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.031100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.033037 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.033013    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 22:00:31.038533 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.038514    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xnl6p"]
Sep 11 22:00:31.088435 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.088406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkq9v\" (UniqueName: \"kubernetes.io/projected/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-kube-api-access-gkq9v\") pod \"authorino-6d4dfc649f-xnl6p\" (UID: \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\") " pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.088602 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.088449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7p2m\" (UniqueName: \"kubernetes.io/projected/3426216f-e850-4777-a1af-9fa6624d3648-kube-api-access-m7p2m\") pod \"authorino-557f876f69-dw5nh\" (UID: \"3426216f-e850-4777-a1af-9fa6624d3648\") " pod="kuadrant-system/authorino-557f876f69-dw5nh"
Sep 11 22:00:31.088602 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.088507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-tls-cert\") pod \"authorino-6d4dfc649f-xnl6p\" (UID: \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\") " pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.107211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.107183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7p2m\" (UniqueName: \"kubernetes.io/projected/3426216f-e850-4777-a1af-9fa6624d3648-kube-api-access-m7p2m\") pod \"authorino-557f876f69-dw5nh\" (UID: \"3426216f-e850-4777-a1af-9fa6624d3648\") " pod="kuadrant-system/authorino-557f876f69-dw5nh"
Sep 11 22:00:31.135915 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.135826    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xnl6p"]
Sep 11 22:00:31.136084 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:00:31.136064    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gkq9v tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-xnl6p" podUID="e3bc949d-5eb4-4c53-b96c-a6e834c763cc"
Sep 11 22:00:31.189675 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.189620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-tls-cert\") pod \"authorino-6d4dfc649f-xnl6p\" (UID: \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\") " pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.189838 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.189729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkq9v\" (UniqueName: \"kubernetes.io/projected/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-kube-api-access-gkq9v\") pod \"authorino-6d4dfc649f-xnl6p\" (UID: \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\") " pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.192123 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.192100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-tls-cert\") pod \"authorino-6d4dfc649f-xnl6p\" (UID: \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\") " pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.198607 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.198585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkq9v\" (UniqueName: \"kubernetes.io/projected/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-kube-api-access-gkq9v\") pod \"authorino-6d4dfc649f-xnl6p\" (UID: \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\") " pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.323567 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.323533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.324036 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.323537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dw5nh"
Sep 11 22:00:31.331242 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.331221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:31.334067 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.334051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dw5nh"
Sep 11 22:00:31.391008 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.390918    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7p2m\" (UniqueName: \"kubernetes.io/projected/3426216f-e850-4777-a1af-9fa6624d3648-kube-api-access-m7p2m\") pod \"3426216f-e850-4777-a1af-9fa6624d3648\" (UID: \"3426216f-e850-4777-a1af-9fa6624d3648\") "
Sep 11 22:00:31.391008 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.390967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkq9v\" (UniqueName: \"kubernetes.io/projected/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-kube-api-access-gkq9v\") pod \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\" (UID: \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\") "
Sep 11 22:00:31.391008 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.391000    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-tls-cert\") pod \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\" (UID: \"e3bc949d-5eb4-4c53-b96c-a6e834c763cc\") "
Sep 11 22:00:31.393180 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.393137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e3bc949d-5eb4-4c53-b96c-a6e834c763cc" (UID: "e3bc949d-5eb4-4c53-b96c-a6e834c763cc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:00:31.393180 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.393172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3426216f-e850-4777-a1af-9fa6624d3648-kube-api-access-m7p2m" (OuterVolumeSpecName: "kube-api-access-m7p2m") pod "3426216f-e850-4777-a1af-9fa6624d3648" (UID: "3426216f-e850-4777-a1af-9fa6624d3648"). InnerVolumeSpecName "kube-api-access-m7p2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:00:31.393333 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.393205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-kube-api-access-gkq9v" (OuterVolumeSpecName: "kube-api-access-gkq9v") pod "e3bc949d-5eb4-4c53-b96c-a6e834c763cc" (UID: "e3bc949d-5eb4-4c53-b96c-a6e834c763cc"). InnerVolumeSpecName "kube-api-access-gkq9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:00:31.491943 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.491905    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7p2m\" (UniqueName: \"kubernetes.io/projected/3426216f-e850-4777-a1af-9fa6624d3648-kube-api-access-m7p2m\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:00:31.491943 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.491936    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkq9v\" (UniqueName: \"kubernetes.io/projected/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-kube-api-access-gkq9v\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:00:31.491943 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:31.491945    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3bc949d-5eb4-4c53-b96c-a6e834c763cc-tls-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:00:32.328682 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:32.328650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-xnl6p"
Sep 11 22:00:32.329091 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:32.328650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dw5nh"
Sep 11 22:00:32.359586 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:32.359556    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xnl6p"]
Sep 11 22:00:32.364983 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:32.364956    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-xnl6p"]
Sep 11 22:00:32.385187 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:32.385161    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dw5nh"]
Sep 11 22:00:32.389039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:32.389017    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dw5nh"]
Sep 11 22:00:32.862182 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:32.862146    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3426216f-e850-4777-a1af-9fa6624d3648" path="/var/lib/kubelet/pods/3426216f-e850-4777-a1af-9fa6624d3648/volumes"
Sep 11 22:00:32.862429 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:32.862407    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3bc949d-5eb4-4c53-b96c-a6e834c763cc" path="/var/lib/kubelet/pods/e3bc949d-5eb4-4c53-b96c-a6e834c763cc/volumes"
Sep 11 22:00:34.649008 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:34.648973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k7sms"]
Sep 11 22:00:34.649528 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:34.649146    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-k7sms" podUID="3a295379-e7f9-46ad-ab9d-5fdad67cc847" containerName="authorino" containerID="cri-o://b55a5d05ca326eeaf14b09bcd5a9bb10f3fac17092c3339452972415d6505cd8" gracePeriod=30
Sep 11 22:00:35.344732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:35.344699    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a295379-e7f9-46ad-ab9d-5fdad67cc847" containerID="b55a5d05ca326eeaf14b09bcd5a9bb10f3fac17092c3339452972415d6505cd8" exitCode=0
Sep 11 22:00:35.344883 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:35.344740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k7sms" event={"ID":"3a295379-e7f9-46ad-ab9d-5fdad67cc847","Type":"ContainerDied","Data":"b55a5d05ca326eeaf14b09bcd5a9bb10f3fac17092c3339452972415d6505cd8"}
Sep 11 22:00:35.382011 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:35.381987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k7sms"
Sep 11 22:00:35.424343 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:35.424310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbvgh\" (UniqueName: \"kubernetes.io/projected/3a295379-e7f9-46ad-ab9d-5fdad67cc847-kube-api-access-gbvgh\") pod \"3a295379-e7f9-46ad-ab9d-5fdad67cc847\" (UID: \"3a295379-e7f9-46ad-ab9d-5fdad67cc847\") "
Sep 11 22:00:35.426185 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:35.426165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a295379-e7f9-46ad-ab9d-5fdad67cc847-kube-api-access-gbvgh" (OuterVolumeSpecName: "kube-api-access-gbvgh") pod "3a295379-e7f9-46ad-ab9d-5fdad67cc847" (UID: "3a295379-e7f9-46ad-ab9d-5fdad67cc847"). InnerVolumeSpecName "kube-api-access-gbvgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:00:35.524954 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:35.524874    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbvgh\" (UniqueName: \"kubernetes.io/projected/3a295379-e7f9-46ad-ab9d-5fdad67cc847-kube-api-access-gbvgh\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:00:36.350874 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:36.350837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k7sms" event={"ID":"3a295379-e7f9-46ad-ab9d-5fdad67cc847","Type":"ContainerDied","Data":"0868939bc622435ac31c1492a038de8d9c732e42b9dee4d376fea68e091c3916"}
Sep 11 22:00:36.350874 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:36.350855    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k7sms"
Sep 11 22:00:36.350874 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:36.350888    2578 scope.go:117] "RemoveContainer" containerID="b55a5d05ca326eeaf14b09bcd5a9bb10f3fac17092c3339452972415d6505cd8"
Sep 11 22:00:36.370873 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:36.370848    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k7sms"]
Sep 11 22:00:36.374848 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:36.374827    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k7sms"]
Sep 11 22:00:36.856108 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:00:36.856075    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a295379-e7f9-46ad-ab9d-5fdad67cc847" path="/var/lib/kubelet/pods/3a295379-e7f9-46ad-ab9d-5fdad67cc847/volumes"
Sep 11 22:01:30.098276 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.098237    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"]
Sep 11 22:01:30.098795 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.098774    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a295379-e7f9-46ad-ab9d-5fdad67cc847" containerName="authorino"
Sep 11 22:01:30.098850 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.098798    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a295379-e7f9-46ad-ab9d-5fdad67cc847" containerName="authorino"
Sep 11 22:01:30.098903 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.098891    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a295379-e7f9-46ad-ab9d-5fdad67cc847" containerName="authorino"
Sep 11 22:01:30.100861 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.100845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.103057 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.103032    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 22:01:30.103569 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.103540    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4wd6\""
Sep 11 22:01:30.103701 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.103582    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 22:01:30.103767 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.103702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 22:01:30.111433 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.111409    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"]
Sep 11 22:01:30.179057 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.179019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.179339 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.179319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.179467 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.179452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9h7\" (UniqueName: \"kubernetes.io/projected/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-kube-api-access-nk9h7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.179560 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.179549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.179687 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.179673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.179848 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.179834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.280593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.280557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.280593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.280596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.280817 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.280626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9h7\" (UniqueName: \"kubernetes.io/projected/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-kube-api-access-nk9h7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.280817 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.280675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.280817 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.280702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.280817 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.280762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.281069 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.281050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.281140 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.281112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.281176 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.281129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.282979 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.282960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.283116 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.283100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.288073 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.288055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9h7\" (UniqueName: \"kubernetes.io/projected/ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2-kube-api-access-nk9h7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6\" (UID: \"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.412013 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.411911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:30.683781 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:30.683745    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"]
Sep 11 22:01:30.686810 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:01:30.686784    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0d7638_f84e_4fa9_a0ea_6ba35e7292e2.slice/crio-e5624cbc519e1d1ca0157f7de884d22c9782010f7b287e04d06b391d3901ebbc WatchSource:0}: Error finding container e5624cbc519e1d1ca0157f7de884d22c9782010f7b287e04d06b391d3901ebbc: Status 404 returned error can't find the container with id e5624cbc519e1d1ca0157f7de884d22c9782010f7b287e04d06b391d3901ebbc
Sep 11 22:01:31.624455 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:31.624425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6" event={"ID":"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2","Type":"ContainerStarted","Data":"e5624cbc519e1d1ca0157f7de884d22c9782010f7b287e04d06b391d3901ebbc"}
Sep 11 22:01:32.630652 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:32.630604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6" event={"ID":"ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2","Type":"ContainerStarted","Data":"c11b6d7d516907006d11a43bbc76c57af71846e90a0c79995485fb42f074bec0"}
Sep 11 22:01:32.647247 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:32.647195    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6" podStartSLOduration=1.722912719 podStartE2EDuration="2.647179886s" totalImagesPullingTime="924.267167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:32.645759329 +0000 UTC m=+554.330706075" watchObservedRunningTime="2026-09-11 22:01:32.647179886 +0000 UTC m=+554.332126600"
Sep 11 22:01:40.412499 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:40.412452    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:40.413082 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:40.412535    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:40.425206 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:40.425179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:40.680900 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:40.680864    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6"
Sep 11 22:01:58.566656 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.566597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"]
Sep 11 22:01:58.571476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.571452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.573518 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.573495    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 22:01:58.578516 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.578492    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"]
Sep 11 22:01:58.624839 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.624807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.624839 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.624840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.625038 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.624887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.625038 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.624990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbzc\" (UniqueName: \"kubernetes.io/projected/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-kube-api-access-4mbzc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.625038 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.625023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.625145 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.625052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.725890 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.725854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.725890 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.725895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.726168 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.726029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.726168 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.726088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbzc\" (UniqueName: \"kubernetes.io/projected/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-kube-api-access-4mbzc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.726168 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.726110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.726168 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.726134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.726372 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.726325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.726440 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.726416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.726504 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.726484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.728225 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.728198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.728516 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.728495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.733954 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.733926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbzc\" (UniqueName: \"kubernetes.io/projected/faf1376e-e355-4d21-b60a-31ea3b4ba2dd-kube-api-access-4mbzc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d7lz5\" (UID: \"faf1376e-e355-4d21-b60a-31ea3b4ba2dd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:58.884218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:58.884142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:01:59.157423 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:59.157382    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"]
Sep 11 22:01:59.160203 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:01:59.160170    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf1376e_e355_4d21_b60a_31ea3b4ba2dd.slice/crio-e3f58e63807dd6eae621f19fff98a51728e3b3a2708d00344dfc941649223011 WatchSource:0}: Error finding container e3f58e63807dd6eae621f19fff98a51728e3b3a2708d00344dfc941649223011: Status 404 returned error can't find the container with id e3f58e63807dd6eae621f19fff98a51728e3b3a2708d00344dfc941649223011
Sep 11 22:01:59.764620 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:59.764581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5" event={"ID":"faf1376e-e355-4d21-b60a-31ea3b4ba2dd","Type":"ContainerStarted","Data":"bd43e514a638d67a255bf1bacad9e32ba7757e02ee8790f1954c8e3a73f6c375"}
Sep 11 22:01:59.765027 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:59.764664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5" event={"ID":"faf1376e-e355-4d21-b60a-31ea3b4ba2dd","Type":"ContainerStarted","Data":"e3f58e63807dd6eae621f19fff98a51728e3b3a2708d00344dfc941649223011"}
Sep 11 22:01:59.780913 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:01:59.780855    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5" podStartSLOduration=1.5986241369999998 podStartE2EDuration="1.780835441s" totalImagesPullingTime="182.211304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:59.778859629 +0000 UTC m=+581.463806373" watchObservedRunningTime="2026-09-11 22:01:59.780835441 +0000 UTC m=+581.465782155"
Sep 11 22:02:08.884704 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:08.884662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:02:08.885108 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:08.884734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:02:08.897151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:08.897122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:02:09.822401 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:09.822369    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d7lz5"
Sep 11 22:02:47.388903 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.388829    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"]
Sep 11 22:02:47.392014 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.391994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.394220 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.394199    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 22:02:47.402288 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.402264    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"]
Sep 11 22:02:47.450519 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.450470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40625582-367e-4eb8-8df4-2622edae4c10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.450761 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.450523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.450761 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.450612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6sgs\" (UniqueName: \"kubernetes.io/projected/40625582-367e-4eb8-8df4-2622edae4c10-kube-api-access-x6sgs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.450761 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.450665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.450761 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.450747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.451002 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.450789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.551364 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.551330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6sgs\" (UniqueName: \"kubernetes.io/projected/40625582-367e-4eb8-8df4-2622edae4c10-kube-api-access-x6sgs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.551567 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.551377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.551567 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.551418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.551567 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.551442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.551758 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.551656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40625582-367e-4eb8-8df4-2622edae4c10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.551758 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.551727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.551867 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.551794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.551867 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.551808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.552142 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.552083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.553862 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.553832    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40625582-367e-4eb8-8df4-2622edae4c10-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.554216 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.554198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40625582-367e-4eb8-8df4-2622edae4c10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.558562 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.558539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6sgs\" (UniqueName: \"kubernetes.io/projected/40625582-367e-4eb8-8df4-2622edae4c10-kube-api-access-x6sgs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2\" (UID: \"40625582-367e-4eb8-8df4-2622edae4c10\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.703895 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.703857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:47.975893 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.975617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"]
Sep 11 22:02:47.978488 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:02:47.978458    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40625582_367e_4eb8_8df4_2622edae4c10.slice/crio-f1f15ec1ac7df33191af521dc5251896b856f9850a399e624e93ae9484e414c9 WatchSource:0}: Error finding container f1f15ec1ac7df33191af521dc5251896b856f9850a399e624e93ae9484e414c9: Status 404 returned error can't find the container with id f1f15ec1ac7df33191af521dc5251896b856f9850a399e624e93ae9484e414c9
Sep 11 22:02:47.979849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.979830    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:02:47.996584 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:47.996541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2" event={"ID":"40625582-367e-4eb8-8df4-2622edae4c10","Type":"ContainerStarted","Data":"f1f15ec1ac7df33191af521dc5251896b856f9850a399e624e93ae9484e414c9"}
Sep 11 22:02:49.003336 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:49.003296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2" event={"ID":"40625582-367e-4eb8-8df4-2622edae4c10","Type":"ContainerStarted","Data":"43e2f16947af3c16581deff24db04910cf6aced9b22b97ca0fa4ceaf0b10ef20"}
Sep 11 22:02:49.022297 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:49.022244    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2" podStartSLOduration=1.8426311659999999 podStartE2EDuration="2.02222935s" totalImagesPullingTime="179.598184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:02:49.019451355 +0000 UTC m=+630.704398069" watchObservedRunningTime="2026-09-11 22:02:49.02222935 +0000 UTC m=+630.707176063"
Sep 11 22:02:57.704373 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:57.704334    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:57.704905 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:57.704387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:57.716898 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:57.716874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:02:58.061862 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:02:58.061767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2"
Sep 11 22:03:57.078114 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.078020    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"]
Sep 11 22:03:57.081457 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.081435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.083415 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.083395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7e2add-openshift-default-dockercfg-pt2xk\""
Sep 11 22:03:57.094535 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.094510    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"]
Sep 11 22:03:57.127415 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-podinfo\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.127415 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmxdk\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-kube-api-access-kmxdk\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.127684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-certs\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.127684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-token\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.127684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-socket\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.127684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-data\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.127684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-envoy\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.127684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-credential-socket\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.127684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.127628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605bd98e-cb66-4cd4-b7b3-56570708bcea-istiod-ca-cert\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228135 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-socket\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228135 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-data\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228347 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-envoy\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228347 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-credential-socket\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228347 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605bd98e-cb66-4cd4-b7b3-56570708bcea-istiod-ca-cert\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-podinfo\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmxdk\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-kube-api-access-kmxdk\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-certs\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-token\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-data\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-socket\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-credential-socket\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.228865 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.228787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-certs\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.229026 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.229005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605bd98e-cb66-4cd4-b7b3-56570708bcea-istiod-ca-cert\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.230359 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.230339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-envoy\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.230728 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.230708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-podinfo\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.236364 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.236344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-token\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.236555 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.236537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmxdk\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-kube-api-access-kmxdk\") pod \"e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") " pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.391456 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.391377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:57.667775 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.667587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"]
Sep 11 22:03:57.670581 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:03:57.670550    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605bd98e_cb66_4cd4_b7b3_56570708bcea.slice/crio-ec7a933f0b6a246295e413292eafca5d459acce2d92ea49ec09563fa69d0cbb7 WatchSource:0}: Error finding container ec7a933f0b6a246295e413292eafca5d459acce2d92ea49ec09563fa69d0cbb7: Status 404 returned error can't find the container with id ec7a933f0b6a246295e413292eafca5d459acce2d92ea49ec09563fa69d0cbb7
Sep 11 22:03:57.672994 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.672946    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:03:57.673119 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.673033    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:03:57.673119 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:57.673082    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:03:58.353478 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.353445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" event={"ID":"605bd98e-cb66-4cd4-b7b3-56570708bcea","Type":"ContainerStarted","Data":"403038852de2811c2d2a20dadcc46bdba4cf386690e38709e76ecef4f0b49bda"}
Sep 11 22:03:58.353881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.353497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" event={"ID":"605bd98e-cb66-4cd4-b7b3-56570708bcea","Type":"ContainerStarted","Data":"ec7a933f0b6a246295e413292eafca5d459acce2d92ea49ec09563fa69d0cbb7"}
Sep 11 22:03:58.373540 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.373485    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" podStartSLOduration=1.37346781 podStartE2EDuration="1.37346781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:03:58.371162246 +0000 UTC m=+700.056108960" watchObservedRunningTime="2026-09-11 22:03:58.37346781 +0000 UTC m=+700.058414523"
Sep 11 22:03:58.392051 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.391995    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:58.397004 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.396983    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:58.661223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.661136    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-c96d90"]
Sep 11 22:03:58.664565 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.664548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:03:58.666688 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.666664    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h4c6b\""
Sep 11 22:03:58.668549 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.668526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-c96d90"]
Sep 11 22:03:58.740752 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.740714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6qn\" (UniqueName: \"kubernetes.io/projected/b22d7e02-cf1f-4edc-a691-340bb0286978-kube-api-access-nn6qn\") pod \"test-curl-7693-c96d90\" (UID: \"b22d7e02-cf1f-4edc-a691-340bb0286978\") " pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:03:58.841536 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.841490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6qn\" (UniqueName: \"kubernetes.io/projected/b22d7e02-cf1f-4edc-a691-340bb0286978-kube-api-access-nn6qn\") pod \"test-curl-7693-c96d90\" (UID: \"b22d7e02-cf1f-4edc-a691-340bb0286978\") " pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:03:58.849059 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.849038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6qn\" (UniqueName: \"kubernetes.io/projected/b22d7e02-cf1f-4edc-a691-340bb0286978-kube-api-access-nn6qn\") pod \"test-curl-7693-c96d90\" (UID: \"b22d7e02-cf1f-4edc-a691-340bb0286978\") " pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:03:58.975023 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:58.974986    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:03:59.251832 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:59.251806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-c96d90"]
Sep 11 22:03:59.253886 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:03:59.253840    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb22d7e02_cf1f_4edc_a691_340bb0286978.slice/crio-35c2642283d50f888bcfe162fd99d5d31699cdb7de221b0f36a4445b0127a819 WatchSource:0}: Error finding container 35c2642283d50f888bcfe162fd99d5d31699cdb7de221b0f36a4445b0127a819: Status 404 returned error can't find the container with id 35c2642283d50f888bcfe162fd99d5d31699cdb7de221b0f36a4445b0127a819
Sep 11 22:03:59.360011 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:59.359972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-c96d90" event={"ID":"b22d7e02-cf1f-4edc-a691-340bb0286978","Type":"ContainerStarted","Data":"35c2642283d50f888bcfe162fd99d5d31699cdb7de221b0f36a4445b0127a819"}
Sep 11 22:03:59.360465 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:59.360297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:03:59.361225 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:03:59.361205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:04:02.380190 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:02.380150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-c96d90" event={"ID":"b22d7e02-cf1f-4edc-a691-340bb0286978","Type":"ContainerStarted","Data":"998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435"}
Sep 11 22:04:02.405014 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:02.404946    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-c96d90" podStartSLOduration=2.289528825 podStartE2EDuration="4.404924966s" totalImagesPullingTime="2.115396141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:02.40281177 +0000 UTC m=+704.087758484" watchObservedRunningTime="2026-09-11 22:04:02.404924966 +0000 UTC m=+704.089871682"
Sep 11 22:04:03.003547 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.003506    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-c96d90"]
Sep 11 22:04:03.007388 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.007347    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-c96d90"]
Sep 11 22:04:03.124006 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.123925    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-668db9"]
Sep 11 22:04:03.124271 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.124259    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b22d7e02-cf1f-4edc-a691-340bb0286978" containerName="test-curl-7693-c96d90"
Sep 11 22:04:03.124318 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.124273    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b22d7e02-cf1f-4edc-a691-340bb0286978" containerName="test-curl-7693-c96d90"
Sep 11 22:04:03.124353 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.124336    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b22d7e02-cf1f-4edc-a691-340bb0286978" containerName="test-curl-7693-c96d90"
Sep 11 22:04:03.127405 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.127388    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:03.130813 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.130782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-668db9"]
Sep 11 22:04:03.183217 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.183182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/0d75eb68-1418-4a55-b9a3-dae943e2b807-kube-api-access-g9h8v\") pod \"test-curl-7693-668db9\" (UID: \"0d75eb68-1418-4a55-b9a3-dae943e2b807\") " pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:03.283763 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.283724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/0d75eb68-1418-4a55-b9a3-dae943e2b807-kube-api-access-g9h8v\") pod \"test-curl-7693-668db9\" (UID: \"0d75eb68-1418-4a55-b9a3-dae943e2b807\") " pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:03.294061 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.294026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/0d75eb68-1418-4a55-b9a3-dae943e2b807-kube-api-access-g9h8v\") pod \"test-curl-7693-668db9\" (UID: \"0d75eb68-1418-4a55-b9a3-dae943e2b807\") " pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:03.385474 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:03.385392    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-c96d90" container="test-curl-7693-c96d90"
Sep 11 22:04:03.438158 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.438122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:03.712947 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:03.712918    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-668db9"]
Sep 11 22:04:03.715171 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:03.715144    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d75eb68_1418_4a55_b9a3_dae943e2b807.slice/crio-1863e9310cc4be02468aeed5aa8b34d63f4a183cc4640ac5713d59f9c8f54ab6 WatchSource:0}: Error finding container 1863e9310cc4be02468aeed5aa8b34d63f4a183cc4640ac5713d59f9c8f54ab6: Status 404 returned error can't find the container with id 1863e9310cc4be02468aeed5aa8b34d63f4a183cc4640ac5713d59f9c8f54ab6
Sep 11 22:04:04.391777 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.391726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-668db9" event={"ID":"0d75eb68-1418-4a55-b9a3-dae943e2b807","Type":"ContainerStarted","Data":"75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d"}
Sep 11 22:04:04.392220 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.391784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-668db9" event={"ID":"0d75eb68-1418-4a55-b9a3-dae943e2b807","Type":"ContainerStarted","Data":"1863e9310cc4be02468aeed5aa8b34d63f4a183cc4640ac5713d59f9c8f54ab6"}
Sep 11 22:04:04.392220 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.391753    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-c96d90" podUID="b22d7e02-cf1f-4edc-a691-340bb0286978" containerName="test-curl-7693-c96d90" containerID="cri-o://998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435" gracePeriod=2
Sep 11 22:04:04.409800 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:04.409756    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-c96d90\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="b22d7e02-cf1f-4edc-a691-340bb0286978" pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:04:04.409972 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.409833    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-668db9" podStartSLOduration=1.409818341 podStartE2EDuration="1.409818341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:04.408340623 +0000 UTC m=+706.093287336" watchObservedRunningTime="2026-09-11 22:04:04.409818341 +0000 UTC m=+706.094765054"
Sep 11 22:04:04.411221 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:04.411199    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-c96d90\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="b22d7e02-cf1f-4edc-a691-340bb0286978" pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:04:04.439308 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.439277    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"]
Sep 11 22:04:04.442734 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.442711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:04.444592 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:04.444567    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-c96d90\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="b22d7e02-cf1f-4edc-a691-340bb0286978" pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:04:04.444818 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.444802    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-27dad9\""
Sep 11 22:04:04.444880 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.444836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-27dad9-dockercfg-95lkc\""
Sep 11 22:04:04.451662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.451625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"]
Sep 11 22:04:04.492227 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.492190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtdk\" (UniqueName: \"kubernetes.io/projected/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-kube-api-access-kjtdk\") pod \"payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt\" (UID: \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:04.492371 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.492245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt\" (UID: \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:04.593357 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.593327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt\" (UID: \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:04.593510 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.593412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtdk\" (UniqueName: \"kubernetes.io/projected/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-kube-api-access-kjtdk\") pod \"payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt\" (UID: \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:04.594080 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.594052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt\" (UID: \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:04.600821 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.600794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtdk\" (UniqueName: \"kubernetes.io/projected/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-kube-api-access-kjtdk\") pod \"payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt\" (UID: \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:04.754456 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:04.754417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:05.007986 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.007902    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-668db9"]
Sep 11 22:04:05.010744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.010714    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-668db9"]
Sep 11 22:04:05.034419 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.034392    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"]
Sep 11 22:04:05.037211 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:05.037177    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062d34a1_3bf3_4b2f_8dbb_e9cc79130d8e.slice/crio-e1eed7865e3a2e0e7c1c4d6fc25c135670465386c9652fb251163707c8484bb8 WatchSource:0}: Error finding container e1eed7865e3a2e0e7c1c4d6fc25c135670465386c9652fb251163707c8484bb8: Status 404 returned error can't find the container with id e1eed7865e3a2e0e7c1c4d6fc25c135670465386c9652fb251163707c8484bb8
Sep 11 22:04:05.398139 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.398042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt" event={"ID":"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e","Type":"ContainerStarted","Data":"e1eed7865e3a2e0e7c1c4d6fc25c135670465386c9652fb251163707c8484bb8"}
Sep 11 22:04:05.398498 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:05.398202    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-668db9" container="test-curl-7693-668db9"
Sep 11 22:04:05.675203 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.675169    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"]
Sep 11 22:04:05.675707 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.675685    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d75eb68-1418-4a55-b9a3-dae943e2b807" containerName="test-curl-7693-668db9"
Sep 11 22:04:05.675707 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.675706    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d75eb68-1418-4a55-b9a3-dae943e2b807" containerName="test-curl-7693-668db9"
Sep 11 22:04:05.675882 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.675809    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d75eb68-1418-4a55-b9a3-dae943e2b807" containerName="test-curl-7693-668db9"
Sep 11 22:04:05.679082 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.679061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.681217 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.681195    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f823e45b-openshift-default-dockercfg-8lk67\""
Sep 11 22:04:05.689372 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.689347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"]
Sep 11 22:04:05.705078 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-credential-socket\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.705234 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rng2\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-kube-api-access-9rng2\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.705234 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-data\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.705234 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-certs\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.705471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-istio-token\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.705471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083d3498-7ad7-477d-8722-af357659e364-istiod-ca-cert\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.705471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083d3498-7ad7-477d-8722-af357659e364-istio-podinfo\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.705471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-socket\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.705471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.705438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-envoy\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806160 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-credential-socket\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806364 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rng2\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-kube-api-access-9rng2\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806364 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-data\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806364 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-certs\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806364 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-istio-token\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083d3498-7ad7-477d-8722-af357659e364-istiod-ca-cert\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083d3498-7ad7-477d-8722-af357659e364-istio-podinfo\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-socket\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-envoy\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806880 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-credential-socket\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.806880 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-certs\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.807016 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.806993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-data\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.807070 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.807033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-socket\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.807559 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.807534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083d3498-7ad7-477d-8722-af357659e364-istiod-ca-cert\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.809140 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.809114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-envoy\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.809428 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.809405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083d3498-7ad7-477d-8722-af357659e364-istio-podinfo\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.820210 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.820178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rng2\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-kube-api-access-9rng2\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.820611 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.820589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-istio-token\") pod \"e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") " pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:05.835799 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.835767    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"]
Sep 11 22:04:05.840587 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.840565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.842840 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.842813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7d5ec8-openshift-default-dockercfg-6bl62\""
Sep 11 22:04:05.853930 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.853885    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"]
Sep 11 22:04:05.908117 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-socket\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.908301 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-envoy\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.908301 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-podinfo\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.908301 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-token\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.908301 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f45681-2cb0-4ce4-98be-1c231989a113-istiod-ca-cert\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.908301 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-credential-socket\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.908567 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-data\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.908567 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87s6\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-kube-api-access-w87s6\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.908567 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.908395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-certs\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:05.995281 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:05.995202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:06.009619 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-socket\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.009759 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-envoy\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.009759 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-podinfo\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.009759 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-token\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.009928 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f45681-2cb0-4ce4-98be-1c231989a113-istiod-ca-cert\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.009928 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-credential-socket\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.009928 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-data\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.010066 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87s6\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-kube-api-access-w87s6\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.010066 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.009960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-socket\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.010165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.010137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-certs\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.010301 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.010272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-credential-socket\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.010530 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.010503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-certs\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.010663 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.010563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-data\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.010663 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.010587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f45681-2cb0-4ce4-98be-1c231989a113-istiod-ca-cert\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.011981 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.011961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-envoy\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.012298 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.012280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-podinfo\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.017177 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.017153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87s6\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-kube-api-access-w87s6\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.018188 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.018166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-token\") pod \"e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") " pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.154995 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.154951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:06.405138 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.405102    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-668db9" podUID="0d75eb68-1418-4a55-b9a3-dae943e2b807" containerName="test-curl-7693-668db9" containerID="cri-o://75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d" gracePeriod=2
Sep 11 22:04:06.406863 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:06.406834    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-668db9\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0d75eb68-1418-4a55-b9a3-dae943e2b807" pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:06.543387 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.543360    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:04:06.545039 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:06.545012    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-668db9\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0d75eb68-1418-4a55-b9a3-dae943e2b807" pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:06.581682 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.581621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"]
Sep 11 22:04:06.584191 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:06.584158    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f45681_2cb0_4ce4_98be_1c231989a113.slice/crio-b7cf8b180b72ceda7b1c0461f28fb59ff2c5bf3602bfccc47ec5ce599e23230f WatchSource:0}: Error finding container b7cf8b180b72ceda7b1c0461f28fb59ff2c5bf3602bfccc47ec5ce599e23230f: Status 404 returned error can't find the container with id b7cf8b180b72ceda7b1c0461f28fb59ff2c5bf3602bfccc47ec5ce599e23230f
Sep 11 22:04:06.584489 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.584468    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"]
Sep 11 22:04:06.587089 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.587051    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:06.587191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.587125    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:06.587191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.587171    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:06.587656 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:06.587610    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083d3498_7ad7_477d_8722_af357659e364.slice/crio-607fc46aacf08636e10d058dcc75198c4ab51a61057ff34e165263f77990a9d5 WatchSource:0}: Error finding container 607fc46aacf08636e10d058dcc75198c4ab51a61057ff34e165263f77990a9d5: Status 404 returned error can't find the container with id 607fc46aacf08636e10d058dcc75198c4ab51a61057ff34e165263f77990a9d5
Sep 11 22:04:06.589895 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.589834    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:06.589966 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.589923    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:06.589966 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.589950    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:06.616245 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.616222    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn6qn\" (UniqueName: \"kubernetes.io/projected/b22d7e02-cf1f-4edc-a691-340bb0286978-kube-api-access-nn6qn\") pod \"b22d7e02-cf1f-4edc-a691-340bb0286978\" (UID: \"b22d7e02-cf1f-4edc-a691-340bb0286978\") "
Sep 11 22:04:06.618198 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.618173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22d7e02-cf1f-4edc-a691-340bb0286978-kube-api-access-nn6qn" (OuterVolumeSpecName: "kube-api-access-nn6qn") pod "b22d7e02-cf1f-4edc-a691-340bb0286978" (UID: "b22d7e02-cf1f-4edc-a691-340bb0286978"). InnerVolumeSpecName "kube-api-access-nn6qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:06.717884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.717841    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn6qn\" (UniqueName: \"kubernetes.io/projected/b22d7e02-cf1f-4edc-a691-340bb0286978-kube-api-access-nn6qn\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:06.859322 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:06.859240    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b22d7e02-cf1f-4edc-a691-340bb0286978" path="/var/lib/kubelet/pods/b22d7e02-cf1f-4edc-a691-340bb0286978/volumes"
Sep 11 22:04:07.412733 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.412693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt" event={"ID":"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e","Type":"ContainerStarted","Data":"b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd"}
Sep 11 22:04:07.413205 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.412762    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:07.414440 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.414417    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" event={"ID":"d9f45681-2cb0-4ce4-98be-1c231989a113","Type":"ContainerStarted","Data":"d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824"}
Sep 11 22:04:07.414538 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.414455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" event={"ID":"d9f45681-2cb0-4ce4-98be-1c231989a113","Type":"ContainerStarted","Data":"b7cf8b180b72ceda7b1c0461f28fb59ff2c5bf3602bfccc47ec5ce599e23230f"}
Sep 11 22:04:07.415646 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.415607    2578 generic.go:356] "Generic (PLEG): container finished" podID="b22d7e02-cf1f-4edc-a691-340bb0286978" containerID="998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435" exitCode=137
Sep 11 22:04:07.415769 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.415679    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-c96d90"
Sep 11 22:04:07.415769 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.415683    2578 scope.go:117] "RemoveContainer" containerID="998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435"
Sep 11 22:04:07.417625 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.417599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" event={"ID":"083d3498-7ad7-477d-8722-af357659e364","Type":"ContainerStarted","Data":"8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940"}
Sep 11 22:04:07.417769 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.417682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" event={"ID":"083d3498-7ad7-477d-8722-af357659e364","Type":"ContainerStarted","Data":"607fc46aacf08636e10d058dcc75198c4ab51a61057ff34e165263f77990a9d5"}
Sep 11 22:04:07.425143 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.425122    2578 scope.go:117] "RemoveContainer" containerID="998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435"
Sep 11 22:04:07.425426 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:07.425408    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435\": container with ID starting with 998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435 not found: ID does not exist" containerID="998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435"
Sep 11 22:04:07.425471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.425434    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435"} err="failed to get container status \"998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435\": rpc error: code = NotFound desc = could not find container \"998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435\": container with ID starting with 998bb6497d054456adde5ccff5d33ecbf36e81780152aa4373d00aa287f27435 not found: ID does not exist"
Sep 11 22:04:07.430885 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.430839    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt" podStartSLOduration=2.116579671 podStartE2EDuration="3.430821603s" totalImagesPullingTime="1.314241932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:07.427506841 +0000 UTC m=+709.112453554" watchObservedRunningTime="2026-09-11 22:04:07.430821603 +0000 UTC m=+709.115768322"
Sep 11 22:04:07.452127 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.452051    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" podStartSLOduration=2.452036148 podStartE2EDuration="2.452036148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:07.449782228 +0000 UTC m=+709.134728954" watchObservedRunningTime="2026-09-11 22:04:07.452036148 +0000 UTC m=+709.136982862"
Sep 11 22:04:07.468898 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.468848    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" podStartSLOduration=2.46880785 podStartE2EDuration="2.46880785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:07.466474377 +0000 UTC m=+709.151421091" watchObservedRunningTime="2026-09-11 22:04:07.46880785 +0000 UTC m=+709.153754563"
Sep 11 22:04:07.996312 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:07.996272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:08.001464 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.001439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:08.156138 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.156104    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:08.161628 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.161602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:08.424706 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.424672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:08.425064 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.424736    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:08.425569 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.425547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:08.425717 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.425696    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:08.588475 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.588442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:08.636856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.636817    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/0d75eb68-1418-4a55-b9a3-dae943e2b807-kube-api-access-g9h8v\") pod \"0d75eb68-1418-4a55-b9a3-dae943e2b807\" (UID: \"0d75eb68-1418-4a55-b9a3-dae943e2b807\") "
Sep 11 22:04:08.638837 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.638812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d75eb68-1418-4a55-b9a3-dae943e2b807-kube-api-access-g9h8v" (OuterVolumeSpecName: "kube-api-access-g9h8v") pod "0d75eb68-1418-4a55-b9a3-dae943e2b807" (UID: "0d75eb68-1418-4a55-b9a3-dae943e2b807"). InnerVolumeSpecName "kube-api-access-g9h8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:08.738299 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.738266    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/0d75eb68-1418-4a55-b9a3-dae943e2b807-kube-api-access-g9h8v\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:08.857858 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:08.857816    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d75eb68-1418-4a55-b9a3-dae943e2b807" path="/var/lib/kubelet/pods/0d75eb68-1418-4a55-b9a3-dae943e2b807/volumes"
Sep 11 22:04:09.430999 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:09.430962    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d75eb68-1418-4a55-b9a3-dae943e2b807" containerID="75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d" exitCode=137
Sep 11 22:04:09.431404 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:09.431005    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-668db9"
Sep 11 22:04:09.431404 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:09.431068    2578 scope.go:117] "RemoveContainer" containerID="75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d"
Sep 11 22:04:09.439932 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:09.439701    2578 scope.go:117] "RemoveContainer" containerID="75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d"
Sep 11 22:04:09.439985 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:09.439962    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d\": container with ID starting with 75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d not found: ID does not exist" containerID="75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d"
Sep 11 22:04:09.440022 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:09.439985    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d"} err="failed to get container status \"75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d\": rpc error: code = NotFound desc = could not find container \"75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d\": container with ID starting with 75efe467967392c2e4848cfbe701ce5543439aa3e5ceec7fc66f332927469c9d not found: ID does not exist"
Sep 11 22:04:15.828471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:15.828380    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"]
Sep 11 22:04:15.831966 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:15.831947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:15.834074 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:15.834051    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f823e45b-dockercfg-qhxtg\""
Sep 11 22:04:15.834199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:15.834051    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f823e45b\""
Sep 11 22:04:15.842387 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:15.842363    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"]
Sep 11 22:04:15.909037 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:15.908994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vwkn\" (UniqueName: \"kubernetes.io/projected/efe0b40b-9850-4818-8d0a-e11021819190-kube-api-access-9vwkn\") pod \"payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9\" (UID: \"efe0b40b-9850-4818-8d0a-e11021819190\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:15.909246 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:15.909108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe0b40b-9850-4818-8d0a-e11021819190-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9\" (UID: \"efe0b40b-9850-4818-8d0a-e11021819190\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:16.000689 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.000653    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"]
Sep 11 22:04:16.004138 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.004113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.006162 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.006136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-37182a-openshift-default-dockercfg-vfrcf\""
Sep 11 22:04:16.010091 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.010044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vwkn\" (UniqueName: \"kubernetes.io/projected/efe0b40b-9850-4818-8d0a-e11021819190-kube-api-access-9vwkn\") pod \"payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9\" (UID: \"efe0b40b-9850-4818-8d0a-e11021819190\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:16.010211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.010104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe0b40b-9850-4818-8d0a-e11021819190-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9\" (UID: \"efe0b40b-9850-4818-8d0a-e11021819190\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:16.011042 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.011018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe0b40b-9850-4818-8d0a-e11021819190-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9\" (UID: \"efe0b40b-9850-4818-8d0a-e11021819190\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:16.017453 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.017425    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"]
Sep 11 22:04:16.021529 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.021497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vwkn\" (UniqueName: \"kubernetes.io/projected/efe0b40b-9850-4818-8d0a-e11021819190-kube-api-access-9vwkn\") pod \"payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9\" (UID: \"efe0b40b-9850-4818-8d0a-e11021819190\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:16.111608 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.111507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-credential-socket\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.111608 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.111580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-data\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.111922 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.111614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-podinfo\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.111922 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.111750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-envoy\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.111922 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.111805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrv8\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-kube-api-access-xfrv8\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.111922 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.111838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-token\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.111922 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.111863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852388bd-f055-4887-95b3-9c83b2d4bb83-istiod-ca-cert\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.112093 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.111969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-certs\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.112093 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.112044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-socket\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.145099 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.145060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.212913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-certs\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.213009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-socket\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.213053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-credential-socket\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.213099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-data\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.213133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-podinfo\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.213186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-envoy\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.213221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrv8\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-kube-api-access-xfrv8\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.213248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-token\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.213921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.213273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852388bd-f055-4887-95b3-9c83b2d4bb83-istiod-ca-cert\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.214500 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.214105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852388bd-f055-4887-95b3-9c83b2d4bb83-istiod-ca-cert\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.214500 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.214434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-socket\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.219315 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.215029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-data\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.219315 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.215105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-credential-socket\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.219315 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.215332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-certs\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.222350 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.221175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-podinfo\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.230250 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.224752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-envoy\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.236680 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.231589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrv8\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-kube-api-access-xfrv8\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.238292 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.238259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-token\") pod \"e2e-inf-b-37182a-openshift-default-77c66888d-qr5th\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") " pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.316503 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.316451    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:16.428019 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.427990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"]
Sep 11 22:04:16.429095 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:16.429069    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe0b40b_9850_4818_8d0a_e11021819190.slice/crio-e7380976237278d325ec882db5a8e1c0d5295f3f4d39ff6070b112efe84da9e6 WatchSource:0}: Error finding container e7380976237278d325ec882db5a8e1c0d5295f3f4d39ff6070b112efe84da9e6: Status 404 returned error can't find the container with id e7380976237278d325ec882db5a8e1c0d5295f3f4d39ff6070b112efe84da9e6
Sep 11 22:04:16.497381 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.497347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9" event={"ID":"efe0b40b-9850-4818-8d0a-e11021819190","Type":"ContainerStarted","Data":"e7380976237278d325ec882db5a8e1c0d5295f3f4d39ff6070b112efe84da9e6"}
Sep 11 22:04:16.600253 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.600205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"]
Sep 11 22:04:16.601339 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:16.601304    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852388bd_f055_4887_95b3_9c83b2d4bb83.slice/crio-b2cb7b2f082a668392b99d957a1b22123a02fd7687ab19e0743aba7c297261b5 WatchSource:0}: Error finding container b2cb7b2f082a668392b99d957a1b22123a02fd7687ab19e0743aba7c297261b5: Status 404 returned error can't find the container with id b2cb7b2f082a668392b99d957a1b22123a02fd7687ab19e0743aba7c297261b5
Sep 11 22:04:16.603764 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.603728    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:16.603850 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.603808    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:16.603850 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:16.603837    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:17.505223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:17.505183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" event={"ID":"852388bd-f055-4887-95b3-9c83b2d4bb83","Type":"ContainerStarted","Data":"fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a"}
Sep 11 22:04:17.505752 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:17.505253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" event={"ID":"852388bd-f055-4887-95b3-9c83b2d4bb83","Type":"ContainerStarted","Data":"b2cb7b2f082a668392b99d957a1b22123a02fd7687ab19e0743aba7c297261b5"}
Sep 11 22:04:17.507347 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:17.507303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9" event={"ID":"efe0b40b-9850-4818-8d0a-e11021819190","Type":"ContainerStarted","Data":"f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd"}
Sep 11 22:04:17.507477 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:17.507449    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:17.528757 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:17.528694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" podStartSLOduration=2.52866378 podStartE2EDuration="2.52866378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:17.52477082 +0000 UTC m=+719.209717533" watchObservedRunningTime="2026-09-11 22:04:17.52866378 +0000 UTC m=+719.213610494"
Sep 11 22:04:17.541082 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:17.541030    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9" podStartSLOduration=2.541015566 podStartE2EDuration="2.541015566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:17.538422982 +0000 UTC m=+719.223369713" watchObservedRunningTime="2026-09-11 22:04:17.541015566 +0000 UTC m=+719.225962328"
Sep 11 22:04:18.316923 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:18.316880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:18.321797 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:18.321768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:18.425914 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:18.425885    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:04:18.515519 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:18.515481    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:18.516763 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:18.516724    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:19.523350 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:19.523303    2578 generic.go:356] "Generic (PLEG): container finished" podID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerID="403038852de2811c2d2a20dadcc46bdba4cf386690e38709e76ecef4f0b49bda" exitCode=0
Sep 11 22:04:19.524198 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:19.524168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" event={"ID":"605bd98e-cb66-4cd4-b7b3-56570708bcea","Type":"ContainerDied","Data":"403038852de2811c2d2a20dadcc46bdba4cf386690e38709e76ecef4f0b49bda"}
Sep 11 22:04:19.524654 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:19.524616    2578 scope.go:117] "RemoveContainer" containerID="403038852de2811c2d2a20dadcc46bdba4cf386690e38709e76ecef4f0b49bda"
Sep 11 22:04:19.526524 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:19.526144    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:19.526524 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:19.526248    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:19.526524 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:19.526299    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:20.535867 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:20.535831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" event={"ID":"605bd98e-cb66-4cd4-b7b3-56570708bcea","Type":"ContainerStarted","Data":"eff66ff9177e6c97dd3a8eb2fc0fd6f136d00d1d817a40682d2915fae1c098e3"}
Sep 11 22:04:21.392517 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:21.392483    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:04:21.392831 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:21.392537    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 22:04:21.393008 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:21.392975    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 22:04:22.392728 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:22.392235    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 22:04:22.392728 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:22.392304    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 22:04:23.392684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:23.392620    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 22:04:23.392995 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:23.392964    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 22:04:24.392500 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:24.392450    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 22:04:24.392711 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:24.392524    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 22:04:25.392466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.392358    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 22:04:25.392466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.392411    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 22:04:25.411697 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.411534    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"]
Sep 11 22:04:25.418397 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.418376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:25.420652 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.420615    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-37182a\""
Sep 11 22:04:25.420788 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.420620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-37182a-dockercfg-gqg6x\""
Sep 11 22:04:25.426033 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.426004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"]
Sep 11 22:04:25.491276 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.491225    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"]
Sep 11 22:04:25.500354 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.500319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:25.505703 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.505313    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"]
Sep 11 22:04:25.518795 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.518762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvg5\" (UniqueName: \"kubernetes.io/projected/f825b3ee-7f54-4421-932c-edcaf1d0ff00-kube-api-access-nvvg5\") pod \"payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq\" (UID: \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:25.518957 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.518837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f825b3ee-7f54-4421-932c-edcaf1d0ff00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq\" (UID: \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:25.620113 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.620073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvg5\" (UniqueName: \"kubernetes.io/projected/f825b3ee-7f54-4421-932c-edcaf1d0ff00-kube-api-access-nvvg5\") pod \"payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq\" (UID: \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:25.620263 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.620130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdbz\" (UniqueName: \"kubernetes.io/projected/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-kube-api-access-wmdbz\") pod \"payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj\" (UID: \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:25.620263 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.620202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f825b3ee-7f54-4421-932c-edcaf1d0ff00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq\" (UID: \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:25.620263 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.620228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj\" (UID: \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:25.621190 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.621163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f825b3ee-7f54-4421-932c-edcaf1d0ff00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq\" (UID: \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:25.631593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.631560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvg5\" (UniqueName: \"kubernetes.io/projected/f825b3ee-7f54-4421-932c-edcaf1d0ff00-kube-api-access-nvvg5\") pod \"payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq\" (UID: \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:25.721674 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.721613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj\" (UID: \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:25.721885 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.721760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdbz\" (UniqueName: \"kubernetes.io/projected/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-kube-api-access-wmdbz\") pod \"payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj\" (UID: \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:25.722692 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.722594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj\" (UID: \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:25.729729 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.729696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdbz\" (UniqueName: \"kubernetes.io/projected/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-kube-api-access-wmdbz\") pod \"payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj\" (UID: \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:25.773100 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.773059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:25.813425 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:25.813323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:26.072673 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.072184    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"]
Sep 11 22:04:26.075838 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:26.075803    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf825b3ee_7f54_4421_932c_edcaf1d0ff00.slice/crio-97521eb70b85cd2029abda63b0fa25cbc3ca1ab045196fe5145aa5a02fe14874 WatchSource:0}: Error finding container 97521eb70b85cd2029abda63b0fa25cbc3ca1ab045196fe5145aa5a02fe14874: Status 404 returned error can't find the container with id 97521eb70b85cd2029abda63b0fa25cbc3ca1ab045196fe5145aa5a02fe14874
Sep 11 22:04:26.126980 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.126950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"]
Sep 11 22:04:26.133871 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:26.133844    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f25974d_fb78_4609_8d2e_acf7eb7d6bca.slice/crio-e880bd13238d14b3545c1affcfb9fc418305c240fce2b1560f33155331dda628 WatchSource:0}: Error finding container e880bd13238d14b3545c1affcfb9fc418305c240fce2b1560f33155331dda628: Status 404 returned error can't find the container with id e880bd13238d14b3545c1affcfb9fc418305c240fce2b1560f33155331dda628
Sep 11 22:04:26.398272 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.398188    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:04:26.398991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.398942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:04:26.399870 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.399845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:04:26.589513 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.589472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj" event={"ID":"0f25974d-fb78-4609-8d2e-acf7eb7d6bca","Type":"ContainerStarted","Data":"0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b"}
Sep 11 22:04:26.589719 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.589537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj" event={"ID":"0f25974d-fb78-4609-8d2e-acf7eb7d6bca","Type":"ContainerStarted","Data":"e880bd13238d14b3545c1affcfb9fc418305c240fce2b1560f33155331dda628"}
Sep 11 22:04:26.589813 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.589792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:26.592095 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.592060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq" event={"ID":"f825b3ee-7f54-4421-932c-edcaf1d0ff00","Type":"ContainerStarted","Data":"544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88"}
Sep 11 22:04:26.592259 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.592124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:26.592259 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.592149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq" event={"ID":"f825b3ee-7f54-4421-932c-edcaf1d0ff00","Type":"ContainerStarted","Data":"97521eb70b85cd2029abda63b0fa25cbc3ca1ab045196fe5145aa5a02fe14874"}
Sep 11 22:04:26.607238 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.607167    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj" podStartSLOduration=1.607144431 podStartE2EDuration="1.607144431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:26.606073004 +0000 UTC m=+728.291019730" watchObservedRunningTime="2026-09-11 22:04:26.607144431 +0000 UTC m=+728.292091148"
Sep 11 22:04:26.622700 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:26.622620    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq" podStartSLOduration=1.622598441 podStartE2EDuration="1.622598441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:26.620800338 +0000 UTC m=+728.305747052" watchObservedRunningTime="2026-09-11 22:04:26.622598441 +0000 UTC m=+728.307545158"
Sep 11 22:04:28.517497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:28.517417    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:28.606075 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:28.606045    2578 generic.go:356] "Generic (PLEG): container finished" podID="083d3498-7ad7-477d-8722-af357659e364" containerID="8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940" exitCode=0
Sep 11 22:04:28.606229 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:28.606114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" event={"ID":"083d3498-7ad7-477d-8722-af357659e364","Type":"ContainerDied","Data":"8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940"}
Sep 11 22:04:28.606544 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:28.606528    2578 scope.go:117] "RemoveContainer" containerID="8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940"
Sep 11 22:04:28.607682 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:28.607623    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:28.607790 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:28.607739    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:28.607857 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:28.607792    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:29.613747 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.613712    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerID="d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824" exitCode=0
Sep 11 22:04:29.614185 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.613780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" event={"ID":"d9f45681-2cb0-4ce4-98be-1c231989a113","Type":"ContainerDied","Data":"d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824"}
Sep 11 22:04:29.614253 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.614236    2578 scope.go:117] "RemoveContainer" containerID="d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824"
Sep 11 22:04:29.615448 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.615418    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:29.615541 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.615501    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:29.615599 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.615538    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:29.616108 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.616079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" event={"ID":"083d3498-7ad7-477d-8722-af357659e364","Type":"ContainerStarted","Data":"6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1"}
Sep 11 22:04:29.996024 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.995984    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:29.996445 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.996425    2578 patch_prober.go:28] interesting pod/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 22:04:29.996526 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:29.996506    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 22:04:30.625861 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:30.624130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" event={"ID":"d9f45681-2cb0-4ce4-98be-1c231989a113","Type":"ContainerStarted","Data":"c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24"}
Sep 11 22:04:30.995613 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:30.995570    2578 patch_prober.go:28] interesting pod/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 22:04:30.995837 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:30.995667    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 22:04:31.156277 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:31.156232    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:31.156965 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:31.156926    2578 patch_prober.go:28] interesting pod/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 22:04:31.157114 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:31.156990    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 22:04:31.996667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:31.996133    2578 patch_prober.go:28] interesting pod/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 22:04:31.996667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:31.996199    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 22:04:32.155849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:32.155806    2578 patch_prober.go:28] interesting pod/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 22:04:32.156132 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:32.156101    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 22:04:32.995691 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:32.995620    2578 patch_prober.go:28] interesting pod/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 22:04:32.995915 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:32.995718    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 22:04:33.155800 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:33.155756    2578 patch_prober.go:28] interesting pod/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 22:04:33.156307 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:33.155827    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 22:04:34.000070 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:34.000029    2578 patch_prober.go:28] interesting pod/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 22:04:34.000353 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:34.000321    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 22:04:34.156094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:34.156052    2578 patch_prober.go:28] interesting pod/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 22:04:34.156731 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:34.156696    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 22:04:35.110532 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:35.110499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:35.111448 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:35.111431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:35.112159 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:35.112143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:35.155528 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:35.155491    2578 patch_prober.go:28] interesting pod/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 22:04:35.155835 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:35.155804    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 22:04:36.155793 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:36.155696    2578 patch_prober.go:28] interesting pod/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 22:04:36.155793 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:36.155767    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 22:04:37.160998 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:37.160965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:37.161459 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:37.161280    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:37.162284 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:37.162267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:04:37.601012 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:37.600976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:04:37.601174 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:37.601048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:04:39.688907 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:39.688873    2578 generic.go:356] "Generic (PLEG): container finished" podID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerID="fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a" exitCode=0
Sep 11 22:04:39.689329 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:39.688947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" event={"ID":"852388bd-f055-4887-95b3-9c83b2d4bb83","Type":"ContainerDied","Data":"fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a"}
Sep 11 22:04:39.689376 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:39.689325    2578 scope.go:117] "RemoveContainer" containerID="fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a"
Sep 11 22:04:39.690504 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:39.690470    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:39.690562 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:39.690544    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:39.690602 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:39.690579    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:40.698045 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:40.698004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" event={"ID":"852388bd-f055-4887-95b3-9c83b2d4bb83","Type":"ContainerStarted","Data":"1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c"}
Sep 11 22:04:41.317163 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:41.317119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:41.317353 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:41.317331    2578 patch_prober.go:28] interesting pod/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:41.317401 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:41.317383    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 22:04:42.317497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:42.317460    2578 patch_prober.go:28] interesting pod/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:42.317919 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:42.317513    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 22:04:43.317686 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:43.317626    2578 patch_prober.go:28] interesting pod/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:43.318095 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:43.317709    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 22:04:44.317707 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:44.317670    2578 patch_prober.go:28] interesting pod/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:44.318211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:44.317734    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 22:04:45.317510 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:45.317468    2578 patch_prober.go:28] interesting pod/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:45.317709 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:45.317552    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 22:04:46.321786 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:46.321755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:46.322154 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:46.322131    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:46.322862 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:46.322842    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:04:51.020777 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.020741    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"]
Sep 11 22:04:51.021335 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.021016    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9" podUID="efe0b40b-9850-4818-8d0a-e11021819190" containerName="payload-pre-processing" containerID="cri-o://f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd" gracePeriod=30
Sep 11 22:04:51.296741 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.296713    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:51.313491 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.313457    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"]
Sep 11 22:04:51.313759 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.313727    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy" containerID="cri-o://6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1" gracePeriod=30
Sep 11 22:04:51.385910 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.385875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vwkn\" (UniqueName: \"kubernetes.io/projected/efe0b40b-9850-4818-8d0a-e11021819190-kube-api-access-9vwkn\") pod \"efe0b40b-9850-4818-8d0a-e11021819190\" (UID: \"efe0b40b-9850-4818-8d0a-e11021819190\") "
Sep 11 22:04:51.386085 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.385919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe0b40b-9850-4818-8d0a-e11021819190-plugins-config-volume\") pod \"efe0b40b-9850-4818-8d0a-e11021819190\" (UID: \"efe0b40b-9850-4818-8d0a-e11021819190\") "
Sep 11 22:04:51.386307 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.386287    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe0b40b-9850-4818-8d0a-e11021819190-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "efe0b40b-9850-4818-8d0a-e11021819190" (UID: "efe0b40b-9850-4818-8d0a-e11021819190"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:51.388127 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.388101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe0b40b-9850-4818-8d0a-e11021819190-kube-api-access-9vwkn" (OuterVolumeSpecName: "kube-api-access-9vwkn") pod "efe0b40b-9850-4818-8d0a-e11021819190" (UID: "efe0b40b-9850-4818-8d0a-e11021819190"). InnerVolumeSpecName "kube-api-access-9vwkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:51.487301 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.487262    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe0b40b-9850-4818-8d0a-e11021819190-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:51.487301 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.487294    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vwkn\" (UniqueName: \"kubernetes.io/projected/efe0b40b-9850-4818-8d0a-e11021819190-kube-api-access-9vwkn\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:51.757449 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.757409    2578 generic.go:356] "Generic (PLEG): container finished" podID="efe0b40b-9850-4818-8d0a-e11021819190" containerID="f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd" exitCode=0
Sep 11 22:04:51.757659 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.757456    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"
Sep 11 22:04:51.757659 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.757498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9" event={"ID":"efe0b40b-9850-4818-8d0a-e11021819190","Type":"ContainerDied","Data":"f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd"}
Sep 11 22:04:51.757659 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.757564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9" event={"ID":"efe0b40b-9850-4818-8d0a-e11021819190","Type":"ContainerDied","Data":"e7380976237278d325ec882db5a8e1c0d5295f3f4d39ff6070b112efe84da9e6"}
Sep 11 22:04:51.757659 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.757591    2578 scope.go:117] "RemoveContainer" containerID="f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd"
Sep 11 22:04:51.766996 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.766977    2578 scope.go:117] "RemoveContainer" containerID="f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd"
Sep 11 22:04:51.767252 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:51.767231    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd\": container with ID starting with f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd not found: ID does not exist" containerID="f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd"
Sep 11 22:04:51.767317 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.767259    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd"} err="failed to get container status \"f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd\": rpc error: code = NotFound desc = could not find container \"f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd\": container with ID starting with f6b2ba3ff4b6d0a9b720faa43bbf917d1076dee423d6a54c016f5caee19feabd not found: ID does not exist"
Sep 11 22:04:51.778300 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.778278    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"]
Sep 11 22:04:51.783992 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:51.783970    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f823e45b-b5bc654bb-dfnp9"]
Sep 11 22:04:52.339742 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.339702    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"]
Sep 11 22:04:52.340391 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.340368    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe0b40b-9850-4818-8d0a-e11021819190" containerName="payload-pre-processing"
Sep 11 22:04:52.340453 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.340395    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe0b40b-9850-4818-8d0a-e11021819190" containerName="payload-pre-processing"
Sep 11 22:04:52.340525 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.340512    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe0b40b-9850-4818-8d0a-e11021819190" containerName="payload-pre-processing"
Sep 11 22:04:52.345175 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.345156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.347196 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.347176    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0cd94651-gw-openshift-default-dockercfg-4lzjv\""
Sep 11 22:04:52.354266 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.354242    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"]
Sep 11 22:04:52.396327 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-istio-token\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.396484 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-socket\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.396484 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-credential-socket\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.396484 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-envoy\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.396484 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19a4962-8727-41ad-8b06-acb73ff35778-istio-podinfo\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.396675 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-certs\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.396675 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9x4\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-kube-api-access-qw9x4\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.396675 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-data\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.396675 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.396611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19a4962-8727-41ad-8b06-acb73ff35778-istiod-ca-cert\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.497605 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.497573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-istio-token\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.497822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.497614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-socket\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.497822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.497747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-credential-socket\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.497822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.497763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-envoy\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.497822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.497783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19a4962-8727-41ad-8b06-acb73ff35778-istio-podinfo\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.497822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.497823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-certs\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.498074 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.497952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9x4\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-kube-api-access-qw9x4\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.498074 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.498026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-data\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.498074 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.498067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19a4962-8727-41ad-8b06-acb73ff35778-istiod-ca-cert\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.498229 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.498200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-socket\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.498229 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.498220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-credential-socket\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.498396 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.498378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-certs\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.498482 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.498462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-data\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.498742 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.498722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19a4962-8727-41ad-8b06-acb73ff35778-istiod-ca-cert\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.500213 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.500186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-envoy\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.500418 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.500401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19a4962-8727-41ad-8b06-acb73ff35778-istio-podinfo\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.511200 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.511129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-istio-token\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.512125 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.512074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9x4\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-kube-api-access-qw9x4\") pod \"e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") " pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.656886 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.656796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:52.857561 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.857524    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe0b40b-9850-4818-8d0a-e11021819190" path="/var/lib/kubelet/pods/efe0b40b-9850-4818-8d0a-e11021819190/volumes"
Sep 11 22:04:52.933053 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.933023    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"]
Sep 11 22:04:52.934961 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:04:52.934933    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19a4962_8727_41ad_8b06_acb73ff35778.slice/crio-3caea6959f75b48762d77fcaaebea7a5a4f8d56b21a21bbd85f4f0f9a45ba895 WatchSource:0}: Error finding container 3caea6959f75b48762d77fcaaebea7a5a4f8d56b21a21bbd85f4f0f9a45ba895: Status 404 returned error can't find the container with id 3caea6959f75b48762d77fcaaebea7a5a4f8d56b21a21bbd85f4f0f9a45ba895
Sep 11 22:04:52.937346 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.937308    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:52.937410 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.937379    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:52.937447 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:52.937411    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:04:53.771992 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:53.771953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" event={"ID":"e19a4962-8727-41ad-8b06-acb73ff35778","Type":"ContainerStarted","Data":"9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd"}
Sep 11 22:04:53.772474 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:53.772006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" event={"ID":"e19a4962-8727-41ad-8b06-acb73ff35778","Type":"ContainerStarted","Data":"3caea6959f75b48762d77fcaaebea7a5a4f8d56b21a21bbd85f4f0f9a45ba895"}
Sep 11 22:04:53.793007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:53.792955    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" podStartSLOduration=1.792940559 podStartE2EDuration="1.792940559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:53.791325823 +0000 UTC m=+755.476272536" watchObservedRunningTime="2026-09-11 22:04:53.792940559 +0000 UTC m=+755.477887272"
Sep 11 22:04:54.657270 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:54.657235    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:54.662441 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:54.662411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:54.777315 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:54.777283    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:54.778190 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:54.778172    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:04:56.571140 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.571110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:56.639941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.639859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rng2\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-kube-api-access-9rng2\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.639941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.639908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-certs\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.639941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.639933    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-credential-socket\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.640223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.639966    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083d3498-7ad7-477d-8722-af357659e364-istiod-ca-cert\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.640223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640001    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083d3498-7ad7-477d-8722-af357659e364-istio-podinfo\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.640223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-socket\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.640223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-data\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.640223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-envoy\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.640223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-istio-token\") pod \"083d3498-7ad7-477d-8722-af357659e364\" (UID: \"083d3498-7ad7-477d-8722-af357659e364\") "
Sep 11 22:04:56.640525 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640222    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:56.640525 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:56.640670 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640520    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-data" (OuterVolumeSpecName: "istio-data") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:56.640670 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640539    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083d3498-7ad7-477d-8722-af357659e364-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:56.640670 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640624    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.640670 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:56.640670 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.640665    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.642160 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.642134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-kube-api-access-9rng2" (OuterVolumeSpecName: "kube-api-access-9rng2") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "kube-api-access-9rng2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:56.642274 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.642225    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/083d3498-7ad7-477d-8722-af357659e364-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:04:56.642333 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.642322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:56.642441 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.642421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-istio-token" (OuterVolumeSpecName: "istio-token") pod "083d3498-7ad7-477d-8722-af357659e364" (UID: "083d3498-7ad7-477d-8722-af357659e364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:56.741629 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.741591    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083d3498-7ad7-477d-8722-af357659e364-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.741629 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.741623    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.741629 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.741657    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.741882 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.741671    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.741882 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.741685    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rng2\" (UniqueName: \"kubernetes.io/projected/083d3498-7ad7-477d-8722-af357659e364-kube-api-access-9rng2\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.741882 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.741697    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083d3498-7ad7-477d-8722-af357659e364-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.741882 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.741705    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083d3498-7ad7-477d-8722-af357659e364-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:04:56.792852 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.792818    2578 generic.go:356] "Generic (PLEG): container finished" podID="083d3498-7ad7-477d-8722-af357659e364" containerID="6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1" exitCode=0
Sep 11 22:04:56.793029 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.792864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"
Sep 11 22:04:56.793029 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.792905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" event={"ID":"083d3498-7ad7-477d-8722-af357659e364","Type":"ContainerDied","Data":"6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1"}
Sep 11 22:04:56.793029 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.792958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7" event={"ID":"083d3498-7ad7-477d-8722-af357659e364","Type":"ContainerDied","Data":"607fc46aacf08636e10d058dcc75198c4ab51a61057ff34e165263f77990a9d5"}
Sep 11 22:04:56.793029 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.792989    2578 scope.go:117] "RemoveContainer" containerID="6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1"
Sep 11 22:04:56.805283 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.805259    2578 scope.go:117] "RemoveContainer" containerID="8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940"
Sep 11 22:04:56.815688 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.815553    2578 scope.go:117] "RemoveContainer" containerID="6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1"
Sep 11 22:04:56.816289 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:56.816168    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1\": container with ID starting with 6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1 not found: ID does not exist" containerID="6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1"
Sep 11 22:04:56.816289 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.816208    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1"} err="failed to get container status \"6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1\": rpc error: code = NotFound desc = could not find container \"6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1\": container with ID starting with 6d4dce9f1ed0ce67aa7123bcf0e91cebd50ea882c206e3a0353b210892a36da1 not found: ID does not exist"
Sep 11 22:04:56.816289 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.816236    2578 scope.go:117] "RemoveContainer" containerID="8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940"
Sep 11 22:04:56.816589 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:04:56.816567    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940\": container with ID starting with 8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940 not found: ID does not exist" containerID="8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940"
Sep 11 22:04:56.816760 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.816597    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940"} err="failed to get container status \"8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940\": rpc error: code = NotFound desc = could not find container \"8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940\": container with ID starting with 8ca9c18808fdd03200dd65f168ff76ea8c12a17cdfe6ade28a8dd97d4e55c940 not found: ID does not exist"
Sep 11 22:04:56.818356 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.818330    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"]
Sep 11 22:04:56.825314 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.825288    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f823e45b-openshift-default-85c6b9c84b-fgdx7"]
Sep 11 22:04:56.856589 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:04:56.856556    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083d3498-7ad7-477d-8722-af357659e364" path="/var/lib/kubelet/pods/083d3498-7ad7-477d-8722-af357659e364/volumes"
Sep 11 22:05:03.576660 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.575842    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"]
Sep 11 22:05:03.576660 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.576405    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy"
Sep 11 22:05:03.576660 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.576428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy"
Sep 11 22:05:03.576660 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.576474    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy"
Sep 11 22:05:03.576660 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.576484    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy"
Sep 11 22:05:03.576660 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.576615    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy"
Sep 11 22:05:03.576660 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.576629    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="083d3498-7ad7-477d-8722-af357659e364" containerName="istio-proxy"
Sep 11 22:05:03.581748 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.581720    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"]
Sep 11 22:05:03.581891 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.581814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.583949 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.583922    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"kube-root-ca.crt\""
Sep 11 22:05:03.583949 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.583945    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"openshift-service-ca.crt\""
Sep 11 22:05:03.584828 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.584543    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"default-dockercfg-2r4jm\""
Sep 11 22:05:03.584828 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.584655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"simulator-w4-7e2add-kserve-self-signed-certs\""
Sep 11 22:05:03.707834 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.707795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-dshm\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.707834 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.707833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-tmp-dir\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.708054 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.707902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkptf\" (UniqueName: \"kubernetes.io/projected/406dcdd6-0d83-4635-8155-acde6c4d46a0-kube-api-access-zkptf\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.708054 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.707946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-model-cache\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.708054 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.707978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-home\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.708054 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.708010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406dcdd6-0d83-4635-8155-acde6c4d46a0-tls-certs\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.808723 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.808688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-model-cache\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.808864 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.808745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-home\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.808864 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.808779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406dcdd6-0d83-4635-8155-acde6c4d46a0-tls-certs\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.808864 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.808844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-dshm\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.808975 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.808865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-tmp-dir\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.808975 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.808909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkptf\" (UniqueName: \"kubernetes.io/projected/406dcdd6-0d83-4635-8155-acde6c4d46a0-kube-api-access-zkptf\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.809236 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.809210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-home\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.809236 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.809224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-model-cache\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.809330 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.809275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-tmp-dir\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.811183 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.811163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-dshm\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.811474 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.811451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406dcdd6-0d83-4635-8155-acde6c4d46a0-tls-certs\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.817091 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.817067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkptf\" (UniqueName: \"kubernetes.io/projected/406dcdd6-0d83-4635-8155-acde6c4d46a0-kube-api-access-zkptf\") pod \"simulator-w4-7e2add-kserve-75ff79b96b-rvk9k\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") " pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:03.892034 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:03.891941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:04.163670 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:04.163627    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"]
Sep 11 22:05:04.165812 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:05:04.165783    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod406dcdd6_0d83_4635_8155_acde6c4d46a0.slice/crio-9c6f10a3b874f7b77988af3de92bf4c477eee074109f0aa3296a083ebef87444 WatchSource:0}: Error finding container 9c6f10a3b874f7b77988af3de92bf4c477eee074109f0aa3296a083ebef87444: Status 404 returned error can't find the container with id 9c6f10a3b874f7b77988af3de92bf4c477eee074109f0aa3296a083ebef87444
Sep 11 22:05:04.839150 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:04.839108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k" event={"ID":"406dcdd6-0d83-4635-8155-acde6c4d46a0","Type":"ContainerStarted","Data":"d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3"}
Sep 11 22:05:04.839150 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:04.839155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k" event={"ID":"406dcdd6-0d83-4635-8155-acde6c4d46a0","Type":"ContainerStarted","Data":"9c6f10a3b874f7b77988af3de92bf4c477eee074109f0aa3296a083ebef87444"}
Sep 11 22:05:11.166897 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.166825    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k" podStartSLOduration=8.166800858 podStartE2EDuration="8.166800858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:05:04.859074404 +0000 UTC m=+766.544021118" watchObservedRunningTime="2026-09-11 22:05:11.166800858 +0000 UTC m=+772.851747573"
Sep 11 22:05:11.169461 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.169432    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"]
Sep 11 22:05:11.173059 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.173043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.175659 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.175586    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"simulator-w0-95d6c2-kserve-self-signed-certs\""
Sep 11 22:05:11.175659 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.175604    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"default-dockercfg-xs9lg\""
Sep 11 22:05:11.175814 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.175772    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"kube-root-ca.crt\""
Sep 11 22:05:11.175867 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.175852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"openshift-service-ca.crt\""
Sep 11 22:05:11.185182 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.185160    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"]
Sep 11 22:05:11.282644 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.282606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-home\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.282818 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.282667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tmp-dir\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.282818 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.282753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfl4c\" (UniqueName: \"kubernetes.io/projected/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-kube-api-access-hfl4c\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.282818 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.282814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-model-cache\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.283015 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.282917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-dshm\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.283015 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.282955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tls-certs\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384022 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.383988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-home\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384022 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.384026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tmp-dir\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384260 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.384067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfl4c\" (UniqueName: \"kubernetes.io/projected/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-kube-api-access-hfl4c\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384260 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.384121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-model-cache\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384260 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.384172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-dshm\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384260 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.384197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tls-certs\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384474 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.384461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-home\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384536 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.384505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tmp-dir\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.384604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.384585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-model-cache\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.386438 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.386407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-dshm\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.386757 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.386741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tls-certs\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.392341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.392314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfl4c\" (UniqueName: \"kubernetes.io/projected/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-kube-api-access-hfl4c\") pod \"simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") " pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.483193 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.483154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:11.785915 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.785823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"]
Sep 11 22:05:11.789297 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:05:11.789265    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4b84bf8_1509_4eb1_81ea_475d1a1f78a0.slice/crio-d38518574f0e5e9a75a10d267400c4ce8658490d879a48559be38aebbac5a418 WatchSource:0}: Error finding container d38518574f0e5e9a75a10d267400c4ce8658490d879a48559be38aebbac5a418: Status 404 returned error can't find the container with id d38518574f0e5e9a75a10d267400c4ce8658490d879a48559be38aebbac5a418
Sep 11 22:05:11.882232 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.882194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87" event={"ID":"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0","Type":"ContainerStarted","Data":"ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf"}
Sep 11 22:05:11.882415 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.882239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87" event={"ID":"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0","Type":"ContainerStarted","Data":"d38518574f0e5e9a75a10d267400c4ce8658490d879a48559be38aebbac5a418"}
Sep 11 22:05:11.924777 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:11.924704    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87" podStartSLOduration=0.924678887 podStartE2EDuration="924.678887ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:05:11.922549996 +0000 UTC m=+773.607496790" watchObservedRunningTime="2026-09-11 22:05:11.924678887 +0000 UTC m=+773.609625602"
Sep 11 22:05:13.892358 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:13.892325    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:13.892930 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:13.892371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:13.906254 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:13.906221    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:14.902376 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:14.902345    2578 generic.go:356] "Generic (PLEG): container finished" podID="e19a4962-8727-41ad-8b06-acb73ff35778" containerID="9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd" exitCode=0
Sep 11 22:05:14.902768 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:14.902411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" event={"ID":"e19a4962-8727-41ad-8b06-acb73ff35778","Type":"ContainerDied","Data":"9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd"}
Sep 11 22:05:14.902894 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:14.902880    2578 scope.go:117] "RemoveContainer" containerID="9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd"
Sep 11 22:05:14.904066 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:14.904038    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:05:14.904144 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:14.904105    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:05:14.904144 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:14.904138    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:05:14.914184 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:14.914163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:05:15.909583 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:15.909543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" event={"ID":"e19a4962-8727-41ad-8b06-acb73ff35778","Type":"ContainerStarted","Data":"4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127"}
Sep 11 22:05:16.657594 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:16.657541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:05:16.657826 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:16.657736    2578 patch_prober.go:28] interesting pod/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 11 22:05:16.657826 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:16.657801    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 11 22:05:17.657937 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:17.657892    2578 patch_prober.go:28] interesting pod/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 11 22:05:17.658365 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:17.657955    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 11 22:05:18.658235 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:18.658190    2578 patch_prober.go:28] interesting pod/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 11 22:05:18.658673 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:18.658264    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 11 22:05:19.657726 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:19.657683    2578 patch_prober.go:28] interesting pod/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 11 22:05:19.657906 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:19.657755    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 11 22:05:20.657827 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:20.657782    2578 patch_prober.go:28] interesting pod/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 11 22:05:20.658240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:20.657851    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 11 22:05:21.484239 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:21.484192    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:21.484449 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:21.484275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:21.496794 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:21.496767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:21.658171 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:21.658133    2578 patch_prober.go:28] interesting pod/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 11 22:05:21.658570 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:21.658204    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 11 22:05:21.965950 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:21.965913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:05:22.662322 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:22.662292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:05:22.662751 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:22.662669    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:05:22.663530 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:22.663513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:05:27.274498 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:27.274456    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"]
Sep 11 22:05:27.274895 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:27.274745    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy" containerID="cri-o://4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127" gracePeriod=30
Sep 11 22:05:32.524942 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.524912    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:05:32.598488 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598414    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-istio-token\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.598488 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598468    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-credential-socket\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.598739 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-envoy\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.598739 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598526    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw9x4\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-kube-api-access-qw9x4\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.598739 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-certs\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.598739 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598628    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19a4962-8727-41ad-8b06-acb73ff35778-istiod-ca-cert\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.598739 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-data\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.599010 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-socket\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.599010 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598779    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19a4962-8727-41ad-8b06-acb73ff35778-istio-podinfo\") pod \"e19a4962-8727-41ad-8b06-acb73ff35778\" (UID: \"e19a4962-8727-41ad-8b06-acb73ff35778\") "
Sep 11 22:05:32.599010 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:32.599010 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.598877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:32.599237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.599036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-data" (OuterVolumeSpecName: "istio-data") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:32.599237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.599046    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:32.599237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.599201    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:32.599237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.599221    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:32.599237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.599236    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:32.599489 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.599249    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:32.599489 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.599309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19a4962-8727-41ad-8b06-acb73ff35778-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:32.600910 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.600877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:32.601056 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.600920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e19a4962-8727-41ad-8b06-acb73ff35778-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:05:32.601056 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.600939    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-kube-api-access-qw9x4" (OuterVolumeSpecName: "kube-api-access-qw9x4") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "kube-api-access-qw9x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:32.601304 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.601288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-istio-token" (OuterVolumeSpecName: "istio-token") pod "e19a4962-8727-41ad-8b06-acb73ff35778" (UID: "e19a4962-8727-41ad-8b06-acb73ff35778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:32.700455 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.700420    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:32.700455 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.700449    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19a4962-8727-41ad-8b06-acb73ff35778-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:32.700455 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.700460    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw9x4\" (UniqueName: \"kubernetes.io/projected/e19a4962-8727-41ad-8b06-acb73ff35778-kube-api-access-qw9x4\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:32.700748 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.700469    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19a4962-8727-41ad-8b06-acb73ff35778-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:32.700748 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:32.700478    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19a4962-8727-41ad-8b06-acb73ff35778-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:05:33.018936 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.018902    2578 generic.go:356] "Generic (PLEG): container finished" podID="e19a4962-8727-41ad-8b06-acb73ff35778" containerID="4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127" exitCode=0
Sep 11 22:05:33.018936 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.018947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" event={"ID":"e19a4962-8727-41ad-8b06-acb73ff35778","Type":"ContainerDied","Data":"4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127"}
Sep 11 22:05:33.019194 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.018962    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"
Sep 11 22:05:33.019194 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.019001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v" event={"ID":"e19a4962-8727-41ad-8b06-acb73ff35778","Type":"ContainerDied","Data":"3caea6959f75b48762d77fcaaebea7a5a4f8d56b21a21bbd85f4f0f9a45ba895"}
Sep 11 22:05:33.019194 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.019030    2578 scope.go:117] "RemoveContainer" containerID="4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127"
Sep 11 22:05:33.031055 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.031026    2578 scope.go:117] "RemoveContainer" containerID="9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd"
Sep 11 22:05:33.035656 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.035608    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"]
Sep 11 22:05:33.040617 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.040591    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0cd94651-gw-openshift-default-bd7fc5848-lkp2v"]
Sep 11 22:05:33.041225 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.041206    2578 scope.go:117] "RemoveContainer" containerID="4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127"
Sep 11 22:05:33.041537 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:05:33.041518    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127\": container with ID starting with 4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127 not found: ID does not exist" containerID="4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127"
Sep 11 22:05:33.041616 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.041548    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127"} err="failed to get container status \"4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127\": rpc error: code = NotFound desc = could not find container \"4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127\": container with ID starting with 4ae5a3167d97bf89c8af596dc6458471a2afc73546007ffa2b0015806208e127 not found: ID does not exist"
Sep 11 22:05:33.041616 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.041574    2578 scope.go:117] "RemoveContainer" containerID="9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd"
Sep 11 22:05:33.041917 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:05:33.041894    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd\": container with ID starting with 9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd not found: ID does not exist" containerID="9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd"
Sep 11 22:05:33.041970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:33.041931    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd"} err="failed to get container status \"9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd\": rpc error: code = NotFound desc = could not find container \"9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd\": container with ID starting with 9ef30ab0751a24921fb1a5be2e637cb3474157b2564826f4e75462448b57efbd not found: ID does not exist"
Sep 11 22:05:34.857371 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:34.857339    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" path="/var/lib/kubelet/pods/e19a4962-8727-41ad-8b06-acb73ff35778/volumes"
Sep 11 22:05:49.291726 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.291684    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"]
Sep 11 22:05:49.292161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.292107    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy"
Sep 11 22:05:49.292161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.292123    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy"
Sep 11 22:05:49.292161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.292134    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy"
Sep 11 22:05:49.292161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.292140    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy"
Sep 11 22:05:49.292300 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.292207    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy"
Sep 11 22:05:49.292300 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.292217    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19a4962-8727-41ad-8b06-acb73ff35778" containerName="istio-proxy"
Sep 11 22:05:49.295409 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.295391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.297960 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.297933    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-27dad9\"/\"default-dockercfg-dk7pd\""
Sep 11 22:05:49.298079 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.297987    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-27dad9\"/\"kube-root-ca.crt\""
Sep 11 22:05:49.298079 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.297933    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-27dad9\"/\"openshift-service-ca.crt\""
Sep 11 22:05:49.298079 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.297946    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-27dad9\"/\"test-model-27dad9-kserve-self-signed-certs\""
Sep 11 22:05:49.303307 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.303287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"]
Sep 11 22:05:49.469431 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.469372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-dshm\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.469431 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.469430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-model-cache\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.469775 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.469562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwzs\" (UniqueName: \"kubernetes.io/projected/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-kube-api-access-8kwzs\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.469775 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.469611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tls-certs\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.469775 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.469700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tmp-dir\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.469775 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.469735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-home\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.570825 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.570721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwzs\" (UniqueName: \"kubernetes.io/projected/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-kube-api-access-8kwzs\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.570825 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.570769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tls-certs\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.570825 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.570796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tmp-dir\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.570825 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.570825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-home\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.571172 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.570916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-dshm\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.571172 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.570938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-model-cache\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.571282 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.571226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tmp-dir\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.571282 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.571265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-home\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.571355 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.571299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-model-cache\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.573194 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.573164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-dshm\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.573474 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.573453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tls-certs\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.578590 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.578566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwzs\" (UniqueName: \"kubernetes.io/projected/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-kube-api-access-8kwzs\") pod \"test-model-27dad9-kserve-65c988d9cf-v7db7\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") " pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.606289 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.606261    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:49.879417 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:49.879391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"]
Sep 11 22:05:49.881475 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:05:49.881444    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6062fdc9_a21f_49aa_94d5_dc8dc978c13e.slice/crio-02d5de039119ca7306b7cc6d36fc1c7a30772125b49fd9feac6651b7e32c123c WatchSource:0}: Error finding container 02d5de039119ca7306b7cc6d36fc1c7a30772125b49fd9feac6651b7e32c123c: Status 404 returned error can't find the container with id 02d5de039119ca7306b7cc6d36fc1c7a30772125b49fd9feac6651b7e32c123c
Sep 11 22:05:50.114704 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:50.114663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7" event={"ID":"6062fdc9-a21f-49aa-94d5-dc8dc978c13e","Type":"ContainerStarted","Data":"a2506d37b4c2eede79be863323b0ba98877f28d599191b342407216a47d97098"}
Sep 11 22:05:50.114704 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:50.114712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7" event={"ID":"6062fdc9-a21f-49aa-94d5-dc8dc978c13e","Type":"ContainerStarted","Data":"02d5de039119ca7306b7cc6d36fc1c7a30772125b49fd9feac6651b7e32c123c"}
Sep 11 22:05:50.132263 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:50.132155    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7" podStartSLOduration=1.132141671 podStartE2EDuration="1.132141671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:05:50.129771552 +0000 UTC m=+811.814718266" watchObservedRunningTime="2026-09-11 22:05:50.132141671 +0000 UTC m=+811.817088384"
Sep 11 22:05:56.386024 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.385985    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"]
Sep 11 22:05:56.389643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.389614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"
Sep 11 22:05:56.391488 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.391466    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:05:56.396055 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.396029    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"]
Sep 11 22:05:56.437448 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.437404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7029e4-8a74-4864-b8e9-3a075c3a303a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7\" (UID: \"bc7029e4-8a74-4864-b8e9-3a075c3a303a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"
Sep 11 22:05:56.538621 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.538574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7029e4-8a74-4864-b8e9-3a075c3a303a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7\" (UID: \"bc7029e4-8a74-4864-b8e9-3a075c3a303a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"
Sep 11 22:05:56.539238 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.539217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7029e4-8a74-4864-b8e9-3a075c3a303a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7\" (UID: \"bc7029e4-8a74-4864-b8e9-3a075c3a303a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"
Sep 11 22:05:56.702979 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.702937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"
Sep 11 22:05:56.971243 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:56.971214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"]
Sep 11 22:05:56.973242 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:05:56.973218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7029e4_8a74_4864_b8e9_3a075c3a303a.slice/crio-6704d8bf45020bfba2a1c19b65abffe5cd1e984438ae06b1d7b8457b6cf82e1d WatchSource:0}: Error finding container 6704d8bf45020bfba2a1c19b65abffe5cd1e984438ae06b1d7b8457b6cf82e1d: Status 404 returned error can't find the container with id 6704d8bf45020bfba2a1c19b65abffe5cd1e984438ae06b1d7b8457b6cf82e1d
Sep 11 22:05:57.157117 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:57.157081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7" event={"ID":"bc7029e4-8a74-4864-b8e9-3a075c3a303a","Type":"ContainerStarted","Data":"6704d8bf45020bfba2a1c19b65abffe5cd1e984438ae06b1d7b8457b6cf82e1d"}
Sep 11 22:05:59.607200 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:59.607128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:59.607200 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:59.607186    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:05:59.622696 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:05:59.622655    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:06:00.176618 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:00.176522    2578 generic.go:356] "Generic (PLEG): container finished" podID="bc7029e4-8a74-4864-b8e9-3a075c3a303a" containerID="3b1be81ba9c0137cd1ae4f454ea0ee0c03518feedd235e75d59db3ad415489cf" exitCode=0
Sep 11 22:06:00.176618 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:00.176600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7" event={"ID":"bc7029e4-8a74-4864-b8e9-3a075c3a303a","Type":"ContainerDied","Data":"3b1be81ba9c0137cd1ae4f454ea0ee0c03518feedd235e75d59db3ad415489cf"}
Sep 11 22:06:00.188395 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:00.188366    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:06:01.310920 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:01.310893    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"
Sep 11 22:06:01.394211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:01.394165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7029e4-8a74-4864-b8e9-3a075c3a303a-maas-api-service-ca\") pod \"bc7029e4-8a74-4864-b8e9-3a075c3a303a\" (UID: \"bc7029e4-8a74-4864-b8e9-3a075c3a303a\") "
Sep 11 22:06:01.394892 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:01.394862    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7029e4-8a74-4864-b8e9-3a075c3a303a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc7029e4-8a74-4864-b8e9-3a075c3a303a" (UID: "bc7029e4-8a74-4864-b8e9-3a075c3a303a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:01.496194 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:01.496111    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7029e4-8a74-4864-b8e9-3a075c3a303a-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:06:02.189610 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:02.189582    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"
Sep 11 22:06:02.189811 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:02.189622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7" event={"ID":"bc7029e4-8a74-4864-b8e9-3a075c3a303a","Type":"ContainerDied","Data":"6704d8bf45020bfba2a1c19b65abffe5cd1e984438ae06b1d7b8457b6cf82e1d"}
Sep 11 22:06:02.189811 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:02.189680    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6704d8bf45020bfba2a1c19b65abffe5cd1e984438ae06b1d7b8457b6cf82e1d"
Sep 11 22:06:13.096814 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:13.096776    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"]
Sep 11 22:06:13.105884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:13.105851    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f399ea1-8f2r7"]
Sep 11 22:06:14.857771 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:14.857738    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7029e4-8a74-4864-b8e9-3a075c3a303a" path="/var/lib/kubelet/pods/bc7029e4-8a74-4864-b8e9-3a075c3a303a/volumes"
Sep 11 22:06:15.674669 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.671566    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"]
Sep 11 22:06:15.674669 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.672511    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7029e4-8a74-4864-b8e9-3a075c3a303a" containerName="revoke-keys"
Sep 11 22:06:15.674669 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.672535    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7029e4-8a74-4864-b8e9-3a075c3a303a" containerName="revoke-keys"
Sep 11 22:06:15.674669 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.672779    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7029e4-8a74-4864-b8e9-3a075c3a303a" containerName="revoke-keys"
Sep 11 22:06:15.676951 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.676913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.679846 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.679383    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7d5ec8\"/\"default-dockercfg-gwcp4\""
Sep 11 22:06:15.680344 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.680319    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7d5ec8\"/\"openshift-service-ca.crt\""
Sep 11 22:06:15.680505 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.680480    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7d5ec8\"/\"autores-model-7d5ec8-kserve-self-signed-certs\""
Sep 11 22:06:15.680588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.680402    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7d5ec8\"/\"kube-root-ca.crt\""
Sep 11 22:06:15.683559 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.683537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"]
Sep 11 22:06:15.745792 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.745751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tmp-dir\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.745969 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.745811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-model-cache\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.745969 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.745887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8kd\" (UniqueName: \"kubernetes.io/projected/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-kube-api-access-4p8kd\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.745969 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.745923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-dshm\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.745969 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.745958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-home\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.746110 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.745975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tls-certs\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.847352 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.847318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8kd\" (UniqueName: \"kubernetes.io/projected/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-kube-api-access-4p8kd\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.847566 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.847412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-dshm\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.847566 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.847470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-home\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.847566 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.847493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tls-certs\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.847766 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.847566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tmp-dir\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.847766 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.847646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-model-cache\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.848035 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.848007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-home\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.848148 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.848052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tmp-dir\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.848148 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.848084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-model-cache\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.850119 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.850095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tls-certs\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.850262 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.850245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-dshm\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.855152 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.855118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8kd\" (UniqueName: \"kubernetes.io/projected/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-kube-api-access-4p8kd\") pod \"autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") " pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:15.988264 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:15.988159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:16.282794 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:16.282613    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"]
Sep 11 22:06:16.285097 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:06:16.285068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c8ed4ea_e273_43d5_8ceb_fa1e219ab826.slice/crio-6a322200067af3572125dbe7937c96a9b354bf00e58648bfc041914d28164020 WatchSource:0}: Error finding container 6a322200067af3572125dbe7937c96a9b354bf00e58648bfc041914d28164020: Status 404 returned error can't find the container with id 6a322200067af3572125dbe7937c96a9b354bf00e58648bfc041914d28164020
Sep 11 22:06:17.275329 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:17.275279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8" event={"ID":"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826","Type":"ContainerStarted","Data":"569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b"}
Sep 11 22:06:17.275762 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:17.275338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8" event={"ID":"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826","Type":"ContainerStarted","Data":"6a322200067af3572125dbe7937c96a9b354bf00e58648bfc041914d28164020"}
Sep 11 22:06:17.297254 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:17.297200    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8" podStartSLOduration=2.297186686 podStartE2EDuration="2.297186686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:06:17.295346094 +0000 UTC m=+838.980292807" watchObservedRunningTime="2026-09-11 22:06:17.297186686 +0000 UTC m=+838.982133400"
Sep 11 22:06:25.988859 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:25.988815    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:25.989336 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:25.988876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:26.001929 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:26.001901    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:26.339963 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:26.339874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:06:47.909709 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:47.909676    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"]
Sep 11 22:06:47.913182 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:47.913162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"
Sep 11 22:06:47.915501 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:47.915477    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:06:47.921297 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:47.921273    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"]
Sep 11 22:06:48.075085 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:48.075053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc131-ce7d-4f2f-b0ca-3fddf7b41154-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd\" (UID: \"92ebc131-ce7d-4f2f-b0ca-3fddf7b41154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"
Sep 11 22:06:48.176017 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:48.175988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc131-ce7d-4f2f-b0ca-3fddf7b41154-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd\" (UID: \"92ebc131-ce7d-4f2f-b0ca-3fddf7b41154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"
Sep 11 22:06:48.176617 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:48.176598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc131-ce7d-4f2f-b0ca-3fddf7b41154-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd\" (UID: \"92ebc131-ce7d-4f2f-b0ca-3fddf7b41154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"
Sep 11 22:06:48.225220 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:48.225188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"
Sep 11 22:06:48.511868 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:48.511841    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"]
Sep 11 22:06:48.513361 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:06:48.513329    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ebc131_ce7d_4f2f_b0ca_3fddf7b41154.slice/crio-a45c1d2926294f411b2fa7caabb40a8dbfb31bdecfdedb01932f15f612ddabd2 WatchSource:0}: Error finding container a45c1d2926294f411b2fa7caabb40a8dbfb31bdecfdedb01932f15f612ddabd2: Status 404 returned error can't find the container with id a45c1d2926294f411b2fa7caabb40a8dbfb31bdecfdedb01932f15f612ddabd2
Sep 11 22:06:49.461234 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:49.461199    2578 generic.go:356] "Generic (PLEG): container finished" podID="92ebc131-ce7d-4f2f-b0ca-3fddf7b41154" containerID="b0aa4e0f72029f125f33960206fa956b7049202e60b11ecc99f23738f9469246" exitCode=0
Sep 11 22:06:49.461657 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:49.461245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd" event={"ID":"92ebc131-ce7d-4f2f-b0ca-3fddf7b41154","Type":"ContainerDied","Data":"b0aa4e0f72029f125f33960206fa956b7049202e60b11ecc99f23738f9469246"}
Sep 11 22:06:49.461657 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:49.461286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd" event={"ID":"92ebc131-ce7d-4f2f-b0ca-3fddf7b41154","Type":"ContainerStarted","Data":"a45c1d2926294f411b2fa7caabb40a8dbfb31bdecfdedb01932f15f612ddabd2"}
Sep 11 22:06:50.598720 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:50.598697    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"
Sep 11 22:06:50.704809 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:50.704767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc131-ce7d-4f2f-b0ca-3fddf7b41154-maas-api-service-ca\") pod \"92ebc131-ce7d-4f2f-b0ca-3fddf7b41154\" (UID: \"92ebc131-ce7d-4f2f-b0ca-3fddf7b41154\") "
Sep 11 22:06:50.705135 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:50.705107    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ebc131-ce7d-4f2f-b0ca-3fddf7b41154-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92ebc131-ce7d-4f2f-b0ca-3fddf7b41154" (UID: "92ebc131-ce7d-4f2f-b0ca-3fddf7b41154"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:50.806524 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:50.806428    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc131-ce7d-4f2f-b0ca-3fddf7b41154-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:06:51.473029 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:51.473000    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"
Sep 11 22:06:51.473029 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:51.473014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd" event={"ID":"92ebc131-ce7d-4f2f-b0ca-3fddf7b41154","Type":"ContainerDied","Data":"a45c1d2926294f411b2fa7caabb40a8dbfb31bdecfdedb01932f15f612ddabd2"}
Sep 11 22:06:51.473246 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:06:51.473059    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45c1d2926294f411b2fa7caabb40a8dbfb31bdecfdedb01932f15f612ddabd2"
Sep 11 22:07:04.084478 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:04.084439    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"]
Sep 11 22:07:04.087094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:04.087070    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2c9db036-nf2wd"]
Sep 11 22:07:04.673233 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:04.673199    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"]
Sep 11 22:07:04.856631 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:04.856598    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ebc131-ce7d-4f2f-b0ca-3fddf7b41154" path="/var/lib/kubelet/pods/92ebc131-ce7d-4f2f-b0ca-3fddf7b41154/volumes"
Sep 11 22:07:19.693042 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.692984    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8" podUID="8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" containerName="main" containerID="cri-o://569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b" gracePeriod=45
Sep 11 22:07:19.943191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.943133    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:07:19.999174 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999140    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-home\") pod \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") "
Sep 11 22:07:19.999351 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tmp-dir\") pod \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") "
Sep 11 22:07:19.999351 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999239    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p8kd\" (UniqueName: \"kubernetes.io/projected/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-kube-api-access-4p8kd\") pod \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") "
Sep 11 22:07:19.999351 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-dshm\") pod \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") "
Sep 11 22:07:19.999518 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999367    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tls-certs\") pod \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") "
Sep 11 22:07:19.999518 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999443    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-model-cache\") pod \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\" (UID: \"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826\") "
Sep 11 22:07:19.999518 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999471    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-home" (OuterVolumeSpecName: "home") pod "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" (UID: "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:19.999518 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999488    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" (UID: "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:19.999762 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" (UID: "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:19.999953 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999929    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:20.000015 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999960    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:20.000015 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:19.999974    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:20.001498 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.001476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-dshm" (OuterVolumeSpecName: "dshm") pod "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" (UID: "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:20.001594 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.001569    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" (UID: "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:07:20.001649 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.001602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-kube-api-access-4p8kd" (OuterVolumeSpecName: "kube-api-access-4p8kd") pod "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" (UID: "8c8ed4ea-e273-43d5-8ceb-fa1e219ab826"). InnerVolumeSpecName "kube-api-access-4p8kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:07:20.101503 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.101463    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:20.101503 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.101499    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p8kd\" (UniqueName: \"kubernetes.io/projected/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-kube-api-access-4p8kd\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:20.101503 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.101510    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:20.642980 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.642939    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" containerID="569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b" exitCode=0
Sep 11 22:07:20.643167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.642988    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"
Sep 11 22:07:20.643167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.643027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8" event={"ID":"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826","Type":"ContainerDied","Data":"569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b"}
Sep 11 22:07:20.643167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.643071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8" event={"ID":"8c8ed4ea-e273-43d5-8ceb-fa1e219ab826","Type":"ContainerDied","Data":"6a322200067af3572125dbe7937c96a9b354bf00e58648bfc041914d28164020"}
Sep 11 22:07:20.643167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.643089    2578 scope.go:117] "RemoveContainer" containerID="569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b"
Sep 11 22:07:20.654573 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.654553    2578 scope.go:117] "RemoveContainer" containerID="569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b"
Sep 11 22:07:20.654926 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:07:20.654902    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b\": container with ID starting with 569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b not found: ID does not exist" containerID="569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b"
Sep 11 22:07:20.654984 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.654935    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b"} err="failed to get container status \"569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b\": rpc error: code = NotFound desc = could not find container \"569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b\": container with ID starting with 569dfaa9e1e736049a03d9eac90f198f3522cf51c8b7936d1542fef91d1b6f0b not found: ID does not exist"
Sep 11 22:07:20.669625 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.669591    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"]
Sep 11 22:07:20.672162 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.672139    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7d5ec8/autores-model-7d5ec8-kserve-68fc4f95f5-dj6b8"]
Sep 11 22:07:20.857273 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:20.857238    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" path="/var/lib/kubelet/pods/8c8ed4ea-e273-43d5-8ceb-fa1e219ab826/volumes"
Sep 11 22:07:41.682648 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:41.682601    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"]
Sep 11 22:07:41.683113 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:41.682877    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy" containerID="cri-o://c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24" gracePeriod=30
Sep 11 22:07:46.937173 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:46.937144    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:07:47.064665 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f45681-2cb0-4ce4-98be-1c231989a113-istiod-ca-cert\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.064665 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-envoy\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.064665 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-token\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.064955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064682    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w87s6\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-kube-api-access-w87s6\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.064955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064705    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-podinfo\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.064955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-credential-socket\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.064955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064761    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-data\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.064955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-certs\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.064955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.064854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-socket\") pod \"d9f45681-2cb0-4ce4-98be-1c231989a113\" (UID: \"d9f45681-2cb0-4ce4-98be-1c231989a113\") "
Sep 11 22:07:47.065285 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065068    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:47.065285 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065084    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:47.065285 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:47.065444 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f45681-2cb0-4ce4-98be-1c231989a113-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:07:47.065444 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:47.065558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065485    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f45681-2cb0-4ce4-98be-1c231989a113-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.065558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065506    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.065558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065522    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.065558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.065536    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.067161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.067128    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:07:47.067161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.067139    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:07:47.067310 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.067172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-kube-api-access-w87s6" (OuterVolumeSpecName: "kube-api-access-w87s6") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "kube-api-access-w87s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:07:47.067367 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.067354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9f45681-2cb0-4ce4-98be-1c231989a113" (UID: "d9f45681-2cb0-4ce4-98be-1c231989a113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:47.166876 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.166844    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.166876 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.166870    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.166876 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.166884    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w87s6\" (UniqueName: \"kubernetes.io/projected/d9f45681-2cb0-4ce4-98be-1c231989a113-kube-api-access-w87s6\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.167102 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.166894    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f45681-2cb0-4ce4-98be-1c231989a113-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.167102 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.166902    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f45681-2cb0-4ce4-98be-1c231989a113-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:07:47.797565 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.797528    2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerID="c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24" exitCode=0
Sep 11 22:07:47.797942 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.797573    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"
Sep 11 22:07:47.797942 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.797613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" event={"ID":"d9f45681-2cb0-4ce4-98be-1c231989a113","Type":"ContainerDied","Data":"c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24"}
Sep 11 22:07:47.797942 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.797695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg" event={"ID":"d9f45681-2cb0-4ce4-98be-1c231989a113","Type":"ContainerDied","Data":"b7cf8b180b72ceda7b1c0461f28fb59ff2c5bf3602bfccc47ec5ce599e23230f"}
Sep 11 22:07:47.797942 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.797726    2578 scope.go:117] "RemoveContainer" containerID="c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24"
Sep 11 22:07:47.808299 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.808196    2578 scope.go:117] "RemoveContainer" containerID="d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824"
Sep 11 22:07:47.817124 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.817104    2578 scope.go:117] "RemoveContainer" containerID="c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24"
Sep 11 22:07:47.817389 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:07:47.817369    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24\": container with ID starting with c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24 not found: ID does not exist" containerID="c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24"
Sep 11 22:07:47.817491 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.817401    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24"} err="failed to get container status \"c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24\": rpc error: code = NotFound desc = could not find container \"c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24\": container with ID starting with c64b03b587936f8b0f43ba8ad0321d357b8d4dca7cf31c9a493140f655144b24 not found: ID does not exist"
Sep 11 22:07:47.817491 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.817427    2578 scope.go:117] "RemoveContainer" containerID="d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824"
Sep 11 22:07:47.817828 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:07:47.817806    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824\": container with ID starting with d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824 not found: ID does not exist" containerID="d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824"
Sep 11 22:07:47.817912 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.817838    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824"} err="failed to get container status \"d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824\": rpc error: code = NotFound desc = could not find container \"d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824\": container with ID starting with d936e421a86ad721acb60ee53c5c661b60847327f052e906fc0c904250f8a824 not found: ID does not exist"
Sep 11 22:07:47.820917 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.820897    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"]
Sep 11 22:07:47.823266 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:47.823245    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7d5ec8-openshift-default-78b785c986-bvrzg"]
Sep 11 22:07:48.857048 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:48.857012    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" path="/var/lib/kubelet/pods/d9f45681-2cb0-4ce4-98be-1c231989a113/volumes"
Sep 11 22:07:49.492107 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492061    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"]
Sep 11 22:07:49.492497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492481    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" containerName="main"
Sep 11 22:07:49.492558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492500    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" containerName="main"
Sep 11 22:07:49.492558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492511    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ebc131-ce7d-4f2f-b0ca-3fddf7b41154" containerName="revoke-keys"
Sep 11 22:07:49.492558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ebc131-ce7d-4f2f-b0ca-3fddf7b41154" containerName="revoke-keys"
Sep 11 22:07:49.492558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492531    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy"
Sep 11 22:07:49.492558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492538    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy"
Sep 11 22:07:49.492768 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492597    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8ed4ea-e273-43d5-8ceb-fa1e219ab826" containerName="main"
Sep 11 22:07:49.492768 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492603    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy"
Sep 11 22:07:49.492768 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.492609    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ebc131-ce7d-4f2f-b0ca-3fddf7b41154" containerName="revoke-keys"
Sep 11 22:07:49.497146 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.497128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:49.499298 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.499273    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0aea78-dockercfg-d4zqt\""
Sep 11 22:07:49.499409 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.499328    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0aea78\""
Sep 11 22:07:49.503810 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.503781    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"]
Sep 11 22:07:49.588712 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.588671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc78727-d827-4197-a8d2-16a827390fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f\" (UID: \"1fc78727-d827-4197-a8d2-16a827390fab\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:49.588898 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.588842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clw5\" (UniqueName: \"kubernetes.io/projected/1fc78727-d827-4197-a8d2-16a827390fab-kube-api-access-2clw5\") pod \"payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f\" (UID: \"1fc78727-d827-4197-a8d2-16a827390fab\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:49.689897 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.689857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc78727-d827-4197-a8d2-16a827390fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f\" (UID: \"1fc78727-d827-4197-a8d2-16a827390fab\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:49.690094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.689947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clw5\" (UniqueName: \"kubernetes.io/projected/1fc78727-d827-4197-a8d2-16a827390fab-kube-api-access-2clw5\") pod \"payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f\" (UID: \"1fc78727-d827-4197-a8d2-16a827390fab\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:49.690558 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.690538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc78727-d827-4197-a8d2-16a827390fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f\" (UID: \"1fc78727-d827-4197-a8d2-16a827390fab\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:49.697703 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.697679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clw5\" (UniqueName: \"kubernetes.io/projected/1fc78727-d827-4197-a8d2-16a827390fab-kube-api-access-2clw5\") pod \"payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f\" (UID: \"1fc78727-d827-4197-a8d2-16a827390fab\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:49.808997 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:49.808892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:50.086269 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:50.086239    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"]
Sep 11 22:07:50.088108 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:07:50.088081    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc78727_d827_4197_a8d2_16a827390fab.slice/crio-736342f3caef7d14fbf18c6bf0d31d4194dfaf2a9170ee54659e5909bd360e1d WatchSource:0}: Error finding container 736342f3caef7d14fbf18c6bf0d31d4194dfaf2a9170ee54659e5909bd360e1d: Status 404 returned error can't find the container with id 736342f3caef7d14fbf18c6bf0d31d4194dfaf2a9170ee54659e5909bd360e1d
Sep 11 22:07:50.817796 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:50.817755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f" event={"ID":"1fc78727-d827-4197-a8d2-16a827390fab","Type":"ContainerStarted","Data":"809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438"}
Sep 11 22:07:50.818011 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:50.817847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f" event={"ID":"1fc78727-d827-4197-a8d2-16a827390fab","Type":"ContainerStarted","Data":"736342f3caef7d14fbf18c6bf0d31d4194dfaf2a9170ee54659e5909bd360e1d"}
Sep 11 22:07:50.818011 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:50.817906    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:07:50.833701 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:50.833651    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f" podStartSLOduration=1.833624748 podStartE2EDuration="1.833624748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:07:50.832087298 +0000 UTC m=+932.517034025" watchObservedRunningTime="2026-09-11 22:07:50.833624748 +0000 UTC m=+932.518571462"
Sep 11 22:07:54.059018 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.058984    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"]
Sep 11 22:07:54.059488 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.059364    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy"
Sep 11 22:07:54.059488 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.059376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy"
Sep 11 22:07:54.059488 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.059440    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f45681-2cb0-4ce4-98be-1c231989a113" containerName="istio-proxy"
Sep 11 22:07:54.062540 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.062524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.065241 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.065221    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-37182a\"/\"openshift-service-ca.crt\""
Sep 11 22:07:54.065395 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.065373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-37182a\"/\"kube-root-ca.crt\""
Sep 11 22:07:54.065480 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.065460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-37182a\"/\"default-dockercfg-5p26f\""
Sep 11 22:07:54.065540 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.065523    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-37182a\"/\"test-model-37182a-kserve-self-signed-certs\""
Sep 11 22:07:54.070847 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.070822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"]
Sep 11 22:07:54.135490 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.135453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g244d\" (UniqueName: \"kubernetes.io/projected/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-kube-api-access-g244d\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.135713 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.135533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-model-cache\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.135713 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.135560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tls-certs\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.135713 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.135583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-dshm\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.135713 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.135678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-home\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.135911 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.135712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tmp-dir\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.236419 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.236378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g244d\" (UniqueName: \"kubernetes.io/projected/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-kube-api-access-g244d\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.236615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.236442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-model-cache\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.236615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.236461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tls-certs\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.236615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.236480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-dshm\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.236615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.236508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-home\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.236615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.236535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tmp-dir\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.236997 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.236970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-home\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.236997 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.236990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-model-cache\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.237119 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.237076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tmp-dir\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.238887 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.238857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-dshm\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.239219 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.239202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tls-certs\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.243043 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.243019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g244d\" (UniqueName: \"kubernetes.io/projected/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-kube-api-access-g244d\") pod \"test-model-37182a-kserve-778cd96cd9-jf7hg\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") " pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.373666 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.373540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:07:54.646701 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.646674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"]
Sep 11 22:07:54.648972 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:07:54.648939    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbf4ab82_6018_4491_ad27_1f5bf057dc7d.slice/crio-5d9f9c5bf0de28657c2398fa1d7aa02998154c7f1094db811fbb5741368cf75d WatchSource:0}: Error finding container 5d9f9c5bf0de28657c2398fa1d7aa02998154c7f1094db811fbb5741368cf75d: Status 404 returned error can't find the container with id 5d9f9c5bf0de28657c2398fa1d7aa02998154c7f1094db811fbb5741368cf75d
Sep 11 22:07:54.844253 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.844222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg" event={"ID":"bbf4ab82-6018-4491-ad27-1f5bf057dc7d","Type":"ContainerStarted","Data":"9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499"}
Sep 11 22:07:54.844415 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.844262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg" event={"ID":"bbf4ab82-6018-4491-ad27-1f5bf057dc7d","Type":"ContainerStarted","Data":"5d9f9c5bf0de28657c2398fa1d7aa02998154c7f1094db811fbb5741368cf75d"}
Sep 11 22:07:54.862082 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:07:54.862008    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg" podStartSLOduration=0.861984695 podStartE2EDuration="861.984695ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:07:54.858985898 +0000 UTC m=+936.543932612" watchObservedRunningTime="2026-09-11 22:07:54.861984695 +0000 UTC m=+936.546931410"
Sep 11 22:08:01.825947 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:08:01.825901    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:08:04.374134 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:08:04.374099    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:08:04.374540 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:08:04.374189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:08:04.386710 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:08:04.386685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:08:04.915749 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:08:04.915712    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:09:01.578200 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.578168    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"]
Sep 11 22:09:01.581849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.581828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.584847 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.584825    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0aea78\"/\"openshift-service-ca.crt\""
Sep 11 22:09:01.584960 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.584846    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0aea78\"/\"default-dockercfg-c84zf\""
Sep 11 22:09:01.584960 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.584887    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0aea78\"/\"explicit-model-0aea78-kserve-self-signed-certs\""
Sep 11 22:09:01.584960 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.584825    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0aea78\"/\"kube-root-ca.crt\""
Sep 11 22:09:01.593279 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.593255    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"]
Sep 11 22:09:01.702650 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.702609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tmp-dir\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.702819 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.702690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-dshm\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.702819 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.702729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-home\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.702819 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.702759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jp75\" (UniqueName: \"kubernetes.io/projected/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-kube-api-access-8jp75\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.702819 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.702806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-model-cache\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.702959 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.702831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tls-certs\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.803580 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.803533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tmp-dir\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.803820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.803591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-dshm\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.803820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.803629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-home\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.803820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.803694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jp75\" (UniqueName: \"kubernetes.io/projected/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-kube-api-access-8jp75\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.803820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.803729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-model-cache\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.803820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.803759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tls-certs\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.804170 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.804022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tmp-dir\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.804170 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.804058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-model-cache\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.804170 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.804107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-home\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.805889 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.805869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-dshm\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.806240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.806222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tls-certs\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.811812 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.811792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jp75\" (UniqueName: \"kubernetes.io/projected/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-kube-api-access-8jp75\") pod \"explicit-model-0aea78-kserve-585cff9765-rp9fr\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") " pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:01.893224 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:01.893148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:02.169689 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:02.169629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"]
Sep 11 22:09:02.171876 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:09:02.171847    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29b3a038_9a01_4988_8716_ed0e4a9fb4d9.slice/crio-639991369fd1e828e7ae4dc0c0ec8a23f22434181f6103d167d886a1315cf36d WatchSource:0}: Error finding container 639991369fd1e828e7ae4dc0c0ec8a23f22434181f6103d167d886a1315cf36d: Status 404 returned error can't find the container with id 639991369fd1e828e7ae4dc0c0ec8a23f22434181f6103d167d886a1315cf36d
Sep 11 22:09:02.250842 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:02.250810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr" event={"ID":"29b3a038-9a01-4988-8716-ed0e4a9fb4d9","Type":"ContainerStarted","Data":"bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547"}
Sep 11 22:09:02.251004 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:02.250854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr" event={"ID":"29b3a038-9a01-4988-8716-ed0e4a9fb4d9","Type":"ContainerStarted","Data":"639991369fd1e828e7ae4dc0c0ec8a23f22434181f6103d167d886a1315cf36d"}
Sep 11 22:09:02.268204 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:02.268157    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr" podStartSLOduration=1.268141452 podStartE2EDuration="1.268141452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:09:02.267084752 +0000 UTC m=+1003.952031465" watchObservedRunningTime="2026-09-11 22:09:02.268141452 +0000 UTC m=+1003.953088166"
Sep 11 22:09:11.893705 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:11.893660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:11.894156 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:11.893718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:11.906449 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:11.906421    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:12.317943 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:12.317914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:21.938214 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:21.938178    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"]
Sep 11 22:09:21.941531 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:21.941511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"
Sep 11 22:09:21.943626 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:21.943602    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:09:21.949150 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:21.949122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"]
Sep 11 22:09:22.007716 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:22.007674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414e2b6e-3d14-4b87-9b8d-ec1622f6f15b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27f26465-czdg5\" (UID: \"414e2b6e-3d14-4b87-9b8d-ec1622f6f15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"
Sep 11 22:09:22.108286 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:22.108242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414e2b6e-3d14-4b87-9b8d-ec1622f6f15b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27f26465-czdg5\" (UID: \"414e2b6e-3d14-4b87-9b8d-ec1622f6f15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"
Sep 11 22:09:22.108972 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:22.108951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414e2b6e-3d14-4b87-9b8d-ec1622f6f15b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27f26465-czdg5\" (UID: \"414e2b6e-3d14-4b87-9b8d-ec1622f6f15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"
Sep 11 22:09:22.254543 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:22.254448    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"
Sep 11 22:09:22.525077 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:22.525053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"]
Sep 11 22:09:22.527084 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:09:22.527054    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414e2b6e_3d14_4b87_9b8d_ec1622f6f15b.slice/crio-7b05c79d4a420faeadf192e277c6457c93dd5c70fe497bd093afd32177e4e8ea WatchSource:0}: Error finding container 7b05c79d4a420faeadf192e277c6457c93dd5c70fe497bd093afd32177e4e8ea: Status 404 returned error can't find the container with id 7b05c79d4a420faeadf192e277c6457c93dd5c70fe497bd093afd32177e4e8ea
Sep 11 22:09:23.371822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:23.371788    2578 generic.go:356] "Generic (PLEG): container finished" podID="414e2b6e-3d14-4b87-9b8d-ec1622f6f15b" containerID="4cf9b9c074a92aefcf52a7ef82b8b644c361120490dda1e4c6e6b2bc511e5854" exitCode=0
Sep 11 22:09:23.371822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:23.371828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5" event={"ID":"414e2b6e-3d14-4b87-9b8d-ec1622f6f15b","Type":"ContainerDied","Data":"4cf9b9c074a92aefcf52a7ef82b8b644c361120490dda1e4c6e6b2bc511e5854"}
Sep 11 22:09:23.372238 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:23.371865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5" event={"ID":"414e2b6e-3d14-4b87-9b8d-ec1622f6f15b","Type":"ContainerStarted","Data":"7b05c79d4a420faeadf192e277c6457c93dd5c70fe497bd093afd32177e4e8ea"}
Sep 11 22:09:24.510679 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.510651    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"
Sep 11 22:09:24.529971 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.529940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414e2b6e-3d14-4b87-9b8d-ec1622f6f15b-maas-api-service-ca\") pod \"414e2b6e-3d14-4b87-9b8d-ec1622f6f15b\" (UID: \"414e2b6e-3d14-4b87-9b8d-ec1622f6f15b\") "
Sep 11 22:09:24.530335 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.530309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414e2b6e-3d14-4b87-9b8d-ec1622f6f15b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "414e2b6e-3d14-4b87-9b8d-ec1622f6f15b" (UID: "414e2b6e-3d14-4b87-9b8d-ec1622f6f15b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:24.631439 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.631341    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414e2b6e-3d14-4b87-9b8d-ec1622f6f15b-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:24.900216 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.900134    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"]
Sep 11 22:09:24.900552 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.900539    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414e2b6e-3d14-4b87-9b8d-ec1622f6f15b" containerName="revoke-keys"
Sep 11 22:09:24.900597 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.900554    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="414e2b6e-3d14-4b87-9b8d-ec1622f6f15b" containerName="revoke-keys"
Sep 11 22:09:24.900643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.900626    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="414e2b6e-3d14-4b87-9b8d-ec1622f6f15b" containerName="revoke-keys"
Sep 11 22:09:24.903770 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.903748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:24.905789 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.905767    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"e2e-conflict-a-f25d30a7-kserve-self-signed-certs\""
Sep 11 22:09:24.914524 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.914502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"]
Sep 11 22:09:24.934296 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.934266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c6cded-5048-4ca9-9a74-ba1457144420-tls-certs\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:24.934420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.934321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-dshm\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:24.934420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.934340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-home\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:24.934420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.934360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggr76\" (UniqueName: \"kubernetes.io/projected/17c6cded-5048-4ca9-9a74-ba1457144420-kube-api-access-ggr76\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:24.934420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.934387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-model-cache\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:24.934420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:24.934406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-tmp-dir\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.035441 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.035404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-dshm\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.035441 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.035448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-home\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.035746 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.035469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggr76\" (UniqueName: \"kubernetes.io/projected/17c6cded-5048-4ca9-9a74-ba1457144420-kube-api-access-ggr76\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.035746 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.035499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-model-cache\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.035746 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.035519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-tmp-dir\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.035746 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.035584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c6cded-5048-4ca9-9a74-ba1457144420-tls-certs\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.036004 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.035981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-tmp-dir\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.036071 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.035997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-home\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.036119 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.036062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-model-cache\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.037891 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.037865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-dshm\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.038054 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.038037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c6cded-5048-4ca9-9a74-ba1457144420-tls-certs\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.042824 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.042805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggr76\" (UniqueName: \"kubernetes.io/projected/17c6cded-5048-4ca9-9a74-ba1457144420-kube-api-access-ggr76\") pod \"e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.213608 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.213568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:25.388343 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.388309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5" event={"ID":"414e2b6e-3d14-4b87-9b8d-ec1622f6f15b","Type":"ContainerDied","Data":"7b05c79d4a420faeadf192e277c6457c93dd5c70fe497bd093afd32177e4e8ea"}
Sep 11 22:09:25.388530 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.388377    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b05c79d4a420faeadf192e277c6457c93dd5c70fe497bd093afd32177e4e8ea"
Sep 11 22:09:25.388530 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.388314    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"
Sep 11 22:09:25.491035 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:25.490997    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"]
Sep 11 22:09:25.493432 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:09:25.493409    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17c6cded_5048_4ca9_9a74_ba1457144420.slice/crio-8c353ae7f4781dbac6735ebbd1d7a957e394e0ce4d5d53cb4e20281d537e2536 WatchSource:0}: Error finding container 8c353ae7f4781dbac6735ebbd1d7a957e394e0ce4d5d53cb4e20281d537e2536: Status 404 returned error can't find the container with id 8c353ae7f4781dbac6735ebbd1d7a957e394e0ce4d5d53cb4e20281d537e2536
Sep 11 22:09:26.395998 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:26.395966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n" event={"ID":"17c6cded-5048-4ca9-9a74-ba1457144420","Type":"ContainerStarted","Data":"2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb"}
Sep 11 22:09:26.396452 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:26.396005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n" event={"ID":"17c6cded-5048-4ca9-9a74-ba1457144420","Type":"ContainerStarted","Data":"8c353ae7f4781dbac6735ebbd1d7a957e394e0ce4d5d53cb4e20281d537e2536"}
Sep 11 22:09:26.412525 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:26.412472    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n" podStartSLOduration=2.412451733 podStartE2EDuration="2.412451733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:09:26.410907248 +0000 UTC m=+1028.095853976" watchObservedRunningTime="2026-09-11 22:09:26.412451733 +0000 UTC m=+1028.097398443"
Sep 11 22:09:27.670866 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:27.670830    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"]
Sep 11 22:09:27.790957 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:27.790913    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"]
Sep 11 22:09:27.796197 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:27.796161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"
Sep 11 22:09:27.798601 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:27.798510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:09:27.800914 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:27.800881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"]
Sep 11 22:09:27.861262 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:27.861224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9f6f-464d-4889-b330-459281656d28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9\" (UID: \"0f9c9f6f-464d-4889-b330-459281656d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"
Sep 11 22:09:27.962828 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:27.962729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9f6f-464d-4889-b330-459281656d28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9\" (UID: \"0f9c9f6f-464d-4889-b330-459281656d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"
Sep 11 22:09:27.963354 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:27.963334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9f6f-464d-4889-b330-459281656d28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9\" (UID: \"0f9c9f6f-464d-4889-b330-459281656d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"
Sep 11 22:09:28.109220 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:28.109183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"
Sep 11 22:09:28.381013 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:28.380985    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"]
Sep 11 22:09:28.382446 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:09:28.382406    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9c9f6f_464d_4889_b330_459281656d28.slice/crio-7c7fb7055cb2df8b71b4a79d58be008a1a7cbdd5b7458072f6fa581727bbd8f1 WatchSource:0}: Error finding container 7c7fb7055cb2df8b71b4a79d58be008a1a7cbdd5b7458072f6fa581727bbd8f1: Status 404 returned error can't find the container with id 7c7fb7055cb2df8b71b4a79d58be008a1a7cbdd5b7458072f6fa581727bbd8f1
Sep 11 22:09:28.408561 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:28.408533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9" event={"ID":"0f9c9f6f-464d-4889-b330-459281656d28","Type":"ContainerStarted","Data":"7c7fb7055cb2df8b71b4a79d58be008a1a7cbdd5b7458072f6fa581727bbd8f1"}
Sep 11 22:09:29.417160 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:29.417060    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f9c9f6f-464d-4889-b330-459281656d28" containerID="238bb4b9ccdd3cb0e93df71a8aaf229c414f07347d7dedb16affae88f271fb15" exitCode=0
Sep 11 22:09:29.417524 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:29.417154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9" event={"ID":"0f9c9f6f-464d-4889-b330-459281656d28","Type":"ContainerDied","Data":"238bb4b9ccdd3cb0e93df71a8aaf229c414f07347d7dedb16affae88f271fb15"}
Sep 11 22:09:30.550858 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:30.550834    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"
Sep 11 22:09:30.589700 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:30.589664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9f6f-464d-4889-b330-459281656d28-maas-api-service-ca\") pod \"0f9c9f6f-464d-4889-b330-459281656d28\" (UID: \"0f9c9f6f-464d-4889-b330-459281656d28\") "
Sep 11 22:09:30.589999 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:30.589976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9c9f6f-464d-4889-b330-459281656d28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f9c9f6f-464d-4889-b330-459281656d28" (UID: "0f9c9f6f-464d-4889-b330-459281656d28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:30.590118 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:30.590099    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9f6f-464d-4889-b330-459281656d28-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:31.430450 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:31.430417    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"
Sep 11 22:09:31.430679 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:31.430459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9" event={"ID":"0f9c9f6f-464d-4889-b330-459281656d28","Type":"ContainerDied","Data":"7c7fb7055cb2df8b71b4a79d58be008a1a7cbdd5b7458072f6fa581727bbd8f1"}
Sep 11 22:09:31.430679 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:31.430519    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7fb7055cb2df8b71b4a79d58be008a1a7cbdd5b7458072f6fa581727bbd8f1"
Sep 11 22:09:33.193464 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:33.193426    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"]
Sep 11 22:09:33.197700 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:33.197674    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27f26465-czdg5"]
Sep 11 22:09:34.857651 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:34.857605    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="414e2b6e-3d14-4b87-9b8d-ec1622f6f15b" path="/var/lib/kubelet/pods/414e2b6e-3d14-4b87-9b8d-ec1622f6f15b/volumes"
Sep 11 22:09:35.214523 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:35.214476    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:35.214805 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:35.214781    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:35.227051 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:35.227024    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:35.465965 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:35.465874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:09:37.969478 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:37.969443    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"]
Sep 11 22:09:37.969986 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:37.969761    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f" podUID="1fc78727-d827-4197-a8d2-16a827390fab" containerName="payload-pre-processing" containerID="cri-o://809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438" gracePeriod=30
Sep 11 22:09:38.121227 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.121194    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"]
Sep 11 22:09:38.123600 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.123575    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0aea78-bhrt9"]
Sep 11 22:09:38.245453 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.245427    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:09:38.368755 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.368703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2clw5\" (UniqueName: \"kubernetes.io/projected/1fc78727-d827-4197-a8d2-16a827390fab-kube-api-access-2clw5\") pod \"1fc78727-d827-4197-a8d2-16a827390fab\" (UID: \"1fc78727-d827-4197-a8d2-16a827390fab\") "
Sep 11 22:09:38.368947 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.368811    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc78727-d827-4197-a8d2-16a827390fab-plugins-config-volume\") pod \"1fc78727-d827-4197-a8d2-16a827390fab\" (UID: \"1fc78727-d827-4197-a8d2-16a827390fab\") "
Sep 11 22:09:38.369148 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.369127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc78727-d827-4197-a8d2-16a827390fab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1fc78727-d827-4197-a8d2-16a827390fab" (UID: "1fc78727-d827-4197-a8d2-16a827390fab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:38.370946 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.370920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc78727-d827-4197-a8d2-16a827390fab-kube-api-access-2clw5" (OuterVolumeSpecName: "kube-api-access-2clw5") pod "1fc78727-d827-4197-a8d2-16a827390fab" (UID: "1fc78727-d827-4197-a8d2-16a827390fab"). InnerVolumeSpecName "kube-api-access-2clw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:38.470439 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.470350    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2clw5\" (UniqueName: \"kubernetes.io/projected/1fc78727-d827-4197-a8d2-16a827390fab-kube-api-access-2clw5\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:38.470439 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.470382    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc78727-d827-4197-a8d2-16a827390fab-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:38.472941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.472911    2578 generic.go:356] "Generic (PLEG): container finished" podID="1fc78727-d827-4197-a8d2-16a827390fab" containerID="809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438" exitCode=0
Sep 11 22:09:38.473054 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.472959    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"
Sep 11 22:09:38.473054 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.472994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f" event={"ID":"1fc78727-d827-4197-a8d2-16a827390fab","Type":"ContainerDied","Data":"809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438"}
Sep 11 22:09:38.473157 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.473051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f" event={"ID":"1fc78727-d827-4197-a8d2-16a827390fab","Type":"ContainerDied","Data":"736342f3caef7d14fbf18c6bf0d31d4194dfaf2a9170ee54659e5909bd360e1d"}
Sep 11 22:09:38.473157 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.473082    2578 scope.go:117] "RemoveContainer" containerID="809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438"
Sep 11 22:09:38.482421 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.482396    2578 scope.go:117] "RemoveContainer" containerID="809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438"
Sep 11 22:09:38.482788 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:09:38.482763    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438\": container with ID starting with 809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438 not found: ID does not exist" containerID="809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438"
Sep 11 22:09:38.482875 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.482802    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438"} err="failed to get container status \"809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438\": rpc error: code = NotFound desc = could not find container \"809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438\": container with ID starting with 809114454f6686316cb96f5c2dbb5d35aa9c87664579f1ba715b17cf1e82d438 not found: ID does not exist"
Sep 11 22:09:38.493779 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.493750    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"]
Sep 11 22:09:38.500131 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.500100    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aea78-6c7f78999d-pts7f"]
Sep 11 22:09:38.857498 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.857413    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f9c9f6f-464d-4889-b330-459281656d28" path="/var/lib/kubelet/pods/0f9c9f6f-464d-4889-b330-459281656d28/volumes"
Sep 11 22:09:38.857823 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:38.857809    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc78727-d827-4197-a8d2-16a827390fab" path="/var/lib/kubelet/pods/1fc78727-d827-4197-a8d2-16a827390fab/volumes"
Sep 11 22:09:42.690159 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:42.690086    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr" podUID="29b3a038-9a01-4988-8716-ed0e4a9fb4d9" containerName="main" containerID="cri-o://bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547" gracePeriod=45
Sep 11 22:09:42.950344 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:42.950279    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:43.016318 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016285    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-home\") pod \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") "
Sep 11 22:09:43.016468 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jp75\" (UniqueName: \"kubernetes.io/projected/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-kube-api-access-8jp75\") pod \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") "
Sep 11 22:09:43.016468 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016385    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-dshm\") pod \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") "
Sep 11 22:09:43.016570 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016467    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-model-cache\") pod \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") "
Sep 11 22:09:43.016570 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tls-certs\") pod \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") "
Sep 11 22:09:43.016570 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tmp-dir\") pod \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\" (UID: \"29b3a038-9a01-4988-8716-ed0e4a9fb4d9\") "
Sep 11 22:09:43.016570 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-home" (OuterVolumeSpecName: "home") pod "29b3a038-9a01-4988-8716-ed0e4a9fb4d9" (UID: "29b3a038-9a01-4988-8716-ed0e4a9fb4d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:43.016771 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "29b3a038-9a01-4988-8716-ed0e4a9fb4d9" (UID: "29b3a038-9a01-4988-8716-ed0e4a9fb4d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:43.016843 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29b3a038-9a01-4988-8716-ed0e4a9fb4d9" (UID: "29b3a038-9a01-4988-8716-ed0e4a9fb4d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:43.016970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016947    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:43.016970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016967    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:43.017095 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.016977    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:43.018738 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.018708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-kube-api-access-8jp75" (OuterVolumeSpecName: "kube-api-access-8jp75") pod "29b3a038-9a01-4988-8716-ed0e4a9fb4d9" (UID: "29b3a038-9a01-4988-8716-ed0e4a9fb4d9"). InnerVolumeSpecName "kube-api-access-8jp75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:43.018839 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.018747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29b3a038-9a01-4988-8716-ed0e4a9fb4d9" (UID: "29b3a038-9a01-4988-8716-ed0e4a9fb4d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:09:43.018839 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.018750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-dshm" (OuterVolumeSpecName: "dshm") pod "29b3a038-9a01-4988-8716-ed0e4a9fb4d9" (UID: "29b3a038-9a01-4988-8716-ed0e4a9fb4d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:43.117537 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.117493    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:43.117537 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.117530    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jp75\" (UniqueName: \"kubernetes.io/projected/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-kube-api-access-8jp75\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:43.117537 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.117543    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b3a038-9a01-4988-8716-ed0e4a9fb4d9-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:09:43.504993 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.504953    2578 generic.go:356] "Generic (PLEG): container finished" podID="29b3a038-9a01-4988-8716-ed0e4a9fb4d9" containerID="bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547" exitCode=0
Sep 11 22:09:43.504993 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.504997    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"
Sep 11 22:09:43.505286 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.505038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr" event={"ID":"29b3a038-9a01-4988-8716-ed0e4a9fb4d9","Type":"ContainerDied","Data":"bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547"}
Sep 11 22:09:43.505286 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.505083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr" event={"ID":"29b3a038-9a01-4988-8716-ed0e4a9fb4d9","Type":"ContainerDied","Data":"639991369fd1e828e7ae4dc0c0ec8a23f22434181f6103d167d886a1315cf36d"}
Sep 11 22:09:43.505286 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.505098    2578 scope.go:117] "RemoveContainer" containerID="bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547"
Sep 11 22:09:43.514719 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.514701    2578 scope.go:117] "RemoveContainer" containerID="bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547"
Sep 11 22:09:43.515011 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:09:43.514984    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547\": container with ID starting with bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547 not found: ID does not exist" containerID="bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547"
Sep 11 22:09:43.515061 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.515022    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547"} err="failed to get container status \"bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547\": rpc error: code = NotFound desc = could not find container \"bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547\": container with ID starting with bb97fd71afce99117f68e60f10c2cdf73af8620d02dbfa6a48c7e8ef82859547 not found: ID does not exist"
Sep 11 22:09:43.525927 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.525904    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"]
Sep 11 22:09:43.529827 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:43.529804    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0aea78/explicit-model-0aea78-kserve-585cff9765-rp9fr"]
Sep 11 22:09:44.856692 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:44.856621    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b3a038-9a01-4988-8716-ed0e4a9fb4d9" path="/var/lib/kubelet/pods/29b3a038-9a01-4988-8716-ed0e4a9fb4d9/volumes"
Sep 11 22:09:48.062046 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062011    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"]
Sep 11 22:09:48.062414 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062386    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9c9f6f-464d-4889-b330-459281656d28" containerName="revoke-keys"
Sep 11 22:09:48.062414 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062397    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9c9f6f-464d-4889-b330-459281656d28" containerName="revoke-keys"
Sep 11 22:09:48.062414 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062409    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc78727-d827-4197-a8d2-16a827390fab" containerName="payload-pre-processing"
Sep 11 22:09:48.062414 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062416    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc78727-d827-4197-a8d2-16a827390fab" containerName="payload-pre-processing"
Sep 11 22:09:48.062556 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062422    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b3a038-9a01-4988-8716-ed0e4a9fb4d9" containerName="main"
Sep 11 22:09:48.062556 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b3a038-9a01-4988-8716-ed0e4a9fb4d9" containerName="main"
Sep 11 22:09:48.062556 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062482    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9c9f6f-464d-4889-b330-459281656d28" containerName="revoke-keys"
Sep 11 22:09:48.062556 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062499    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fc78727-d827-4197-a8d2-16a827390fab" containerName="payload-pre-processing"
Sep 11 22:09:48.062556 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.062507    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b3a038-9a01-4988-8716-ed0e4a9fb4d9" containerName="main"
Sep 11 22:09:48.065495 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.065472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.067532 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.067506    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"distinct-w0-95d6c2-kserve-self-signed-certs\""
Sep 11 22:09:48.074065 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.074039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"]
Sep 11 22:09:48.167866 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.167822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-tmp-dir\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.167866 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.167872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-model-cache\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.168122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.167912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-dshm\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.168122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.167961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbtb\" (UniqueName: \"kubernetes.io/projected/dbbb5804-1291-49f1-9df9-4496bdc88a42-kube-api-access-ljbtb\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.168122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.167997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-home\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.168122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.168110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbb5804-1291-49f1-9df9-4496bdc88a42-tls-certs\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.269400 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.269359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-tmp-dir\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.269400 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.269402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-model-cache\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.269667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.269432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-dshm\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.269667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.269454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbtb\" (UniqueName: \"kubernetes.io/projected/dbbb5804-1291-49f1-9df9-4496bdc88a42-kube-api-access-ljbtb\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.269667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.269472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-home\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.269834 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.269682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbb5804-1291-49f1-9df9-4496bdc88a42-tls-certs\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.269834 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.269814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-tmp-dir\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.270095 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.270064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-home\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.270214 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.270112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-model-cache\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.271864 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.271838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-dshm\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.272254 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.272233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbb5804-1291-49f1-9df9-4496bdc88a42-tls-certs\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.277419 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.277390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbtb\" (UniqueName: \"kubernetes.io/projected/dbbb5804-1291-49f1-9df9-4496bdc88a42-kube-api-access-ljbtb\") pod \"distinct-w0-95d6c2-kserve-65647494bb-8fqv8\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.376031 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.375935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:48.649984 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:48.649953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"]
Sep 11 22:09:48.651815 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:09:48.651779    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbbb5804_1291_49f1_9df9_4496bdc88a42.slice/crio-5cba850bb8ff77dfdb06e94ef7f59148ccca44d9430f19149a2667b343b5dee3 WatchSource:0}: Error finding container 5cba850bb8ff77dfdb06e94ef7f59148ccca44d9430f19149a2667b343b5dee3: Status 404 returned error can't find the container with id 5cba850bb8ff77dfdb06e94ef7f59148ccca44d9430f19149a2667b343b5dee3
Sep 11 22:09:49.542309 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:49.542265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8" event={"ID":"dbbb5804-1291-49f1-9df9-4496bdc88a42","Type":"ContainerStarted","Data":"e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57"}
Sep 11 22:09:49.542723 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:49.542321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8" event={"ID":"dbbb5804-1291-49f1-9df9-4496bdc88a42","Type":"ContainerStarted","Data":"5cba850bb8ff77dfdb06e94ef7f59148ccca44d9430f19149a2667b343b5dee3"}
Sep 11 22:09:49.558548 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:49.558498    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8" podStartSLOduration=1.558477346 podStartE2EDuration="1.558477346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:09:49.556800831 +0000 UTC m=+1051.241747557" watchObservedRunningTime="2026-09-11 22:09:49.558477346 +0000 UTC m=+1051.243424117"
Sep 11 22:09:58.377035 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:58.376997    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:58.377492 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:58.377258    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:58.391950 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:58.391921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:09:58.613237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:09:58.613208    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:10:28.065521 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.065485    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"]
Sep 11 22:10:28.069131 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.069113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.074766 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.074740    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-38f772\""
Sep 11 22:10:28.074896 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.074836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-38f772-dockercfg-dkrqg\""
Sep 11 22:10:28.085623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.085594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"]
Sep 11 22:10:28.164125 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.164087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl98m\" (UniqueName: \"kubernetes.io/projected/dd3ff945-b7fa-4234-a821-975db5e7ad21-kube-api-access-gl98m\") pod \"payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2\" (UID: \"dd3ff945-b7fa-4234-a821-975db5e7ad21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.164329 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.164195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3ff945-b7fa-4234-a821-975db5e7ad21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2\" (UID: \"dd3ff945-b7fa-4234-a821-975db5e7ad21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.265839 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.265801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl98m\" (UniqueName: \"kubernetes.io/projected/dd3ff945-b7fa-4234-a821-975db5e7ad21-kube-api-access-gl98m\") pod \"payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2\" (UID: \"dd3ff945-b7fa-4234-a821-975db5e7ad21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.266037 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.265905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3ff945-b7fa-4234-a821-975db5e7ad21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2\" (UID: \"dd3ff945-b7fa-4234-a821-975db5e7ad21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.266674 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.266620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3ff945-b7fa-4234-a821-975db5e7ad21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2\" (UID: \"dd3ff945-b7fa-4234-a821-975db5e7ad21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.273234 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.273205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl98m\" (UniqueName: \"kubernetes.io/projected/dd3ff945-b7fa-4234-a821-975db5e7ad21-kube-api-access-gl98m\") pod \"payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2\" (UID: \"dd3ff945-b7fa-4234-a821-975db5e7ad21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.379663 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.379556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.654228 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.654194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"]
Sep 11 22:10:28.656258 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:10:28.656225    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3ff945_b7fa_4234_a821_975db5e7ad21.slice/crio-0e915fd2b7e1edcc23e24e0dcab6bcb8fe689a00eed6c54a23d28fee632a85ac WatchSource:0}: Error finding container 0e915fd2b7e1edcc23e24e0dcab6bcb8fe689a00eed6c54a23d28fee632a85ac: Status 404 returned error can't find the container with id 0e915fd2b7e1edcc23e24e0dcab6bcb8fe689a00eed6c54a23d28fee632a85ac
Sep 11 22:10:28.789129 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.789089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2" event={"ID":"dd3ff945-b7fa-4234-a821-975db5e7ad21","Type":"ContainerStarted","Data":"4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7"}
Sep 11 22:10:28.789341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.789149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2" event={"ID":"dd3ff945-b7fa-4234-a821-975db5e7ad21","Type":"ContainerStarted","Data":"0e915fd2b7e1edcc23e24e0dcab6bcb8fe689a00eed6c54a23d28fee632a85ac"}
Sep 11 22:10:28.789341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.789179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:28.805501 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:28.805399    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2" podStartSLOduration=0.805382561 podStartE2EDuration="805.382561ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:28.804382687 +0000 UTC m=+1090.489329401" watchObservedRunningTime="2026-09-11 22:10:28.805382561 +0000 UTC m=+1090.490329329"
Sep 11 22:10:32.660661 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:32.660596    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"]
Sep 11 22:10:33.888890 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:33.888850    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"]
Sep 11 22:10:33.892471 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:33.892440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"
Sep 11 22:10:33.894649 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:33.894617    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:10:33.900669 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:33.900627    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"]
Sep 11 22:10:34.024797 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:34.024754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044a7e6d-5011-4142-ba96-96dd45f3906b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq\" (UID: \"044a7e6d-5011-4142-ba96-96dd45f3906b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"
Sep 11 22:10:34.126176 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:34.126074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044a7e6d-5011-4142-ba96-96dd45f3906b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq\" (UID: \"044a7e6d-5011-4142-ba96-96dd45f3906b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"
Sep 11 22:10:34.126686 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:34.126665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044a7e6d-5011-4142-ba96-96dd45f3906b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq\" (UID: \"044a7e6d-5011-4142-ba96-96dd45f3906b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"
Sep 11 22:10:34.206522 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:34.206485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"
Sep 11 22:10:34.493734 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:34.493707    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"]
Sep 11 22:10:34.495581 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:10:34.495553    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044a7e6d_5011_4142_ba96_96dd45f3906b.slice/crio-7acfb0f119f27072fec435c8a23fb4c83d59c9356afeac67104366cb6642be7c WatchSource:0}: Error finding container 7acfb0f119f27072fec435c8a23fb4c83d59c9356afeac67104366cb6642be7c: Status 404 returned error can't find the container with id 7acfb0f119f27072fec435c8a23fb4c83d59c9356afeac67104366cb6642be7c
Sep 11 22:10:34.835759 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:34.835726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq" event={"ID":"044a7e6d-5011-4142-ba96-96dd45f3906b","Type":"ContainerStarted","Data":"7acfb0f119f27072fec435c8a23fb4c83d59c9356afeac67104366cb6642be7c"}
Sep 11 22:10:35.842923 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:35.842891    2578 generic.go:356] "Generic (PLEG): container finished" podID="044a7e6d-5011-4142-ba96-96dd45f3906b" containerID="2ee60ba0d9637d34f130d5f07a5dfabe9827a140f68bc96cac22d20f7481fc64" exitCode=0
Sep 11 22:10:35.843372 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:35.842990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq" event={"ID":"044a7e6d-5011-4142-ba96-96dd45f3906b","Type":"ContainerDied","Data":"2ee60ba0d9637d34f130d5f07a5dfabe9827a140f68bc96cac22d20f7481fc64"}
Sep 11 22:10:36.983020 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:36.982992    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"
Sep 11 22:10:37.158991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:37.158896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044a7e6d-5011-4142-ba96-96dd45f3906b-maas-api-service-ca\") pod \"044a7e6d-5011-4142-ba96-96dd45f3906b\" (UID: \"044a7e6d-5011-4142-ba96-96dd45f3906b\") "
Sep 11 22:10:37.159264 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:37.159239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044a7e6d-5011-4142-ba96-96dd45f3906b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "044a7e6d-5011-4142-ba96-96dd45f3906b" (UID: "044a7e6d-5011-4142-ba96-96dd45f3906b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:10:37.260666 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:37.260599    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044a7e6d-5011-4142-ba96-96dd45f3906b-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:37.856786 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:37.856748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq" event={"ID":"044a7e6d-5011-4142-ba96-96dd45f3906b","Type":"ContainerDied","Data":"7acfb0f119f27072fec435c8a23fb4c83d59c9356afeac67104366cb6642be7c"}
Sep 11 22:10:37.856786 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:37.856772    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"
Sep 11 22:10:37.856786 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:37.856796    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7acfb0f119f27072fec435c8a23fb4c83d59c9356afeac67104366cb6642be7c"
Sep 11 22:10:39.797142 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:39.797106    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:10:44.813924 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:44.813887    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"]
Sep 11 22:10:44.814358 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:44.814147    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt" podUID="062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e" containerName="payload-processing" containerID="cri-o://b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd" gracePeriod=30
Sep 11 22:10:44.989116 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:44.989084    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"]
Sep 11 22:10:44.993058 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:44.993027    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27dad9-bb4cq"]
Sep 11 22:10:45.077505 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.077426    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:10:45.138059 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.138021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-plugins-config-volume\") pod \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\" (UID: \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\") "
Sep 11 22:10:45.138219 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.138100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjtdk\" (UniqueName: \"kubernetes.io/projected/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-kube-api-access-kjtdk\") pod \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\" (UID: \"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e\") "
Sep 11 22:10:45.138469 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.138450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e" (UID: "062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:10:45.140181 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.140156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-kube-api-access-kjtdk" (OuterVolumeSpecName: "kube-api-access-kjtdk") pod "062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e" (UID: "062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e"). InnerVolumeSpecName "kube-api-access-kjtdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:45.239112 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.239077    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtdk\" (UniqueName: \"kubernetes.io/projected/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-kube-api-access-kjtdk\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:45.239112 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.239107    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:45.904070 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.904034    2578 generic.go:356] "Generic (PLEG): container finished" podID="062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e" containerID="b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd" exitCode=0
Sep 11 22:10:45.904658 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.904080    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"
Sep 11 22:10:45.904658 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.904123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt" event={"ID":"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e","Type":"ContainerDied","Data":"b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd"}
Sep 11 22:10:45.904658 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.904188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt" event={"ID":"062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e","Type":"ContainerDied","Data":"e1eed7865e3a2e0e7c1c4d6fc25c135670465386c9652fb251163707c8484bb8"}
Sep 11 22:10:45.904658 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.904224    2578 scope.go:117] "RemoveContainer" containerID="b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd"
Sep 11 22:10:45.914155 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.914124    2578 scope.go:117] "RemoveContainer" containerID="b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd"
Sep 11 22:10:45.914439 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:10:45.914419    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd\": container with ID starting with b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd not found: ID does not exist" containerID="b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd"
Sep 11 22:10:45.914500 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.914448    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd"} err="failed to get container status \"b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd\": rpc error: code = NotFound desc = could not find container \"b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd\": container with ID starting with b4daaa24faec4bb1f48e8c44df7344d26ada31af9bc7649bcc6a59e46ee17cdd not found: ID does not exist"
Sep 11 22:10:45.926429 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.926398    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"]
Sep 11 22:10:45.930573 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:45.930545    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27dad9-5ddc58488c-2csqt"]
Sep 11 22:10:46.857996 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:46.857963    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044a7e6d-5011-4142-ba96-96dd45f3906b" path="/var/lib/kubelet/pods/044a7e6d-5011-4142-ba96-96dd45f3906b/volumes"
Sep 11 22:10:46.858323 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:46.858309    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e" path="/var/lib/kubelet/pods/062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e/volumes"
Sep 11 22:10:47.680218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.680174    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7" podUID="6062fdc9-a21f-49aa-94d5-dc8dc978c13e" containerName="main" containerID="cri-o://a2506d37b4c2eede79be863323b0ba98877f28d599191b342407216a47d97098" gracePeriod=45
Sep 11 22:10:47.918950 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.918912    2578 generic.go:356] "Generic (PLEG): container finished" podID="6062fdc9-a21f-49aa-94d5-dc8dc978c13e" containerID="a2506d37b4c2eede79be863323b0ba98877f28d599191b342407216a47d97098" exitCode=0
Sep 11 22:10:47.919143 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.918983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7" event={"ID":"6062fdc9-a21f-49aa-94d5-dc8dc978c13e","Type":"ContainerDied","Data":"a2506d37b4c2eede79be863323b0ba98877f28d599191b342407216a47d97098"}
Sep 11 22:10:47.947672 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.947629    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:10:47.968374 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-model-cache\") pod \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") "
Sep 11 22:10:47.968374 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-home\") pod \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") "
Sep 11 22:10:47.968588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-dshm\") pod \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") "
Sep 11 22:10:47.968588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kwzs\" (UniqueName: \"kubernetes.io/projected/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-kube-api-access-8kwzs\") pod \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") "
Sep 11 22:10:47.968588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968516    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tmp-dir\") pod \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") "
Sep 11 22:10:47.968588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tls-certs\") pod \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\" (UID: \"6062fdc9-a21f-49aa-94d5-dc8dc978c13e\") "
Sep 11 22:10:47.968815 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6062fdc9-a21f-49aa-94d5-dc8dc978c13e" (UID: "6062fdc9-a21f-49aa-94d5-dc8dc978c13e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:47.968815 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968679    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-home" (OuterVolumeSpecName: "home") pod "6062fdc9-a21f-49aa-94d5-dc8dc978c13e" (UID: "6062fdc9-a21f-49aa-94d5-dc8dc978c13e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:47.968923 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6062fdc9-a21f-49aa-94d5-dc8dc978c13e" (UID: "6062fdc9-a21f-49aa-94d5-dc8dc978c13e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:47.968976 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968949    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:47.968976 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968967    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:47.969077 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.968978    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:47.971015 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.970985    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-dshm" (OuterVolumeSpecName: "dshm") pod "6062fdc9-a21f-49aa-94d5-dc8dc978c13e" (UID: "6062fdc9-a21f-49aa-94d5-dc8dc978c13e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:47.971140 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.971062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6062fdc9-a21f-49aa-94d5-dc8dc978c13e" (UID: "6062fdc9-a21f-49aa-94d5-dc8dc978c13e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:10:47.971140 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:47.971081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-kube-api-access-8kwzs" (OuterVolumeSpecName: "kube-api-access-8kwzs") pod "6062fdc9-a21f-49aa-94d5-dc8dc978c13e" (UID: "6062fdc9-a21f-49aa-94d5-dc8dc978c13e"). InnerVolumeSpecName "kube-api-access-8kwzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:48.069600 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:48.069562    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kwzs\" (UniqueName: \"kubernetes.io/projected/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-kube-api-access-8kwzs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:48.069600 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:48.069594    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:48.069600 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:48.069604    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6062fdc9-a21f-49aa-94d5-dc8dc978c13e-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:48.926768 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:48.926738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"
Sep 11 22:10:48.926768 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:48.926746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7" event={"ID":"6062fdc9-a21f-49aa-94d5-dc8dc978c13e","Type":"ContainerDied","Data":"02d5de039119ca7306b7cc6d36fc1c7a30772125b49fd9feac6651b7e32c123c"}
Sep 11 22:10:48.927291 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:48.926800    2578 scope.go:117] "RemoveContainer" containerID="a2506d37b4c2eede79be863323b0ba98877f28d599191b342407216a47d97098"
Sep 11 22:10:48.942151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:48.942120    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"]
Sep 11 22:10:48.945644 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:48.945611    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-27dad9/test-model-27dad9-kserve-65c988d9cf-v7db7"]
Sep 11 22:10:50.857538 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:50.857505    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6062fdc9-a21f-49aa-94d5-dc8dc978c13e" path="/var/lib/kubelet/pods/6062fdc9-a21f-49aa-94d5-dc8dc978c13e/volumes"
Sep 11 22:10:51.643886 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.643851    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"]
Sep 11 22:10:51.644241 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644228    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e" containerName="payload-processing"
Sep 11 22:10:51.644288 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644243    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e" containerName="payload-processing"
Sep 11 22:10:51.644288 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644260    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044a7e6d-5011-4142-ba96-96dd45f3906b" containerName="revoke-keys"
Sep 11 22:10:51.644288 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644265    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="044a7e6d-5011-4142-ba96-96dd45f3906b" containerName="revoke-keys"
Sep 11 22:10:51.644288 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644274    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6062fdc9-a21f-49aa-94d5-dc8dc978c13e" containerName="main"
Sep 11 22:10:51.644288 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644279    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6062fdc9-a21f-49aa-94d5-dc8dc978c13e" containerName="main"
Sep 11 22:10:51.644443 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644344    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="044a7e6d-5011-4142-ba96-96dd45f3906b" containerName="revoke-keys"
Sep 11 22:10:51.644443 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644355    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6062fdc9-a21f-49aa-94d5-dc8dc978c13e" containerName="main"
Sep 11 22:10:51.644443 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.644362    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="062d34a1-3bf3-4b2f-8dbb-e9cc79130d8e" containerName="payload-processing"
Sep 11 22:10:51.648734 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.648712    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"
Sep 11 22:10:51.651818 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.651791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:10:51.663336 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.663300    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"]
Sep 11 22:10:51.704080 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.704030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a80848b-7423-4904-83dd-fec0d859a7db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37182a-mq52g\" (UID: \"5a80848b-7423-4904-83dd-fec0d859a7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"
Sep 11 22:10:51.805695 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.805624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a80848b-7423-4904-83dd-fec0d859a7db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37182a-mq52g\" (UID: \"5a80848b-7423-4904-83dd-fec0d859a7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"
Sep 11 22:10:51.806308 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.806288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a80848b-7423-4904-83dd-fec0d859a7db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-37182a-mq52g\" (UID: \"5a80848b-7423-4904-83dd-fec0d859a7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"
Sep 11 22:10:51.959437 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:51.959403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"
Sep 11 22:10:52.228131 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:52.228105    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"]
Sep 11 22:10:52.229737 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:10:52.229710    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a80848b_7423_4904_83dd_fec0d859a7db.slice/crio-459a015d11251d6f7f7af077fe17956eb940e6feb2866d49ba8cd86ceeb55dd2 WatchSource:0}: Error finding container 459a015d11251d6f7f7af077fe17956eb940e6feb2866d49ba8cd86ceeb55dd2: Status 404 returned error can't find the container with id 459a015d11251d6f7f7af077fe17956eb940e6feb2866d49ba8cd86ceeb55dd2
Sep 11 22:10:52.951671 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:52.951616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g" event={"ID":"5a80848b-7423-4904-83dd-fec0d859a7db","Type":"ContainerStarted","Data":"00e2195419a98b9d082daa575aab309807516d956e81a1e4afea9463d1fae256"}
Sep 11 22:10:52.951844 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:52.951689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g" event={"ID":"5a80848b-7423-4904-83dd-fec0d859a7db","Type":"ContainerStarted","Data":"459a015d11251d6f7f7af077fe17956eb940e6feb2866d49ba8cd86ceeb55dd2"}
Sep 11 22:10:52.966918 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:52.966867    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g" podStartSLOduration=1.966852942 podStartE2EDuration="1.966852942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:52.965598587 +0000 UTC m=+1114.650545301" watchObservedRunningTime="2026-09-11 22:10:52.966852942 +0000 UTC m=+1114.651799655"
Sep 11 22:10:53.958876 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:53.958838    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a80848b-7423-4904-83dd-fec0d859a7db" containerID="00e2195419a98b9d082daa575aab309807516d956e81a1e4afea9463d1fae256" exitCode=0
Sep 11 22:10:53.959059 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:53.958921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g" event={"ID":"5a80848b-7423-4904-83dd-fec0d859a7db","Type":"ContainerDied","Data":"00e2195419a98b9d082daa575aab309807516d956e81a1e4afea9463d1fae256"}
Sep 11 22:10:55.098604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:55.098578    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"
Sep 11 22:10:55.140084 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:55.139950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a80848b-7423-4904-83dd-fec0d859a7db-maas-api-service-ca\") pod \"5a80848b-7423-4904-83dd-fec0d859a7db\" (UID: \"5a80848b-7423-4904-83dd-fec0d859a7db\") "
Sep 11 22:10:55.140420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:55.140395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a80848b-7423-4904-83dd-fec0d859a7db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a80848b-7423-4904-83dd-fec0d859a7db" (UID: "5a80848b-7423-4904-83dd-fec0d859a7db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:10:55.241688 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:55.241583    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a80848b-7423-4904-83dd-fec0d859a7db-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:10:55.975227 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:55.975191    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"
Sep 11 22:10:55.975413 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:55.975228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g" event={"ID":"5a80848b-7423-4904-83dd-fec0d859a7db","Type":"ContainerDied","Data":"459a015d11251d6f7f7af077fe17956eb940e6feb2866d49ba8cd86ceeb55dd2"}
Sep 11 22:10:55.975413 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:10:55.975287    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="459a015d11251d6f7f7af077fe17956eb940e6feb2866d49ba8cd86ceeb55dd2"
Sep 11 22:11:02.303750 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.303713    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"]
Sep 11 22:11:02.304156 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.304113    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a80848b-7423-4904-83dd-fec0d859a7db" containerName="revoke-keys"
Sep 11 22:11:02.304156 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.304124    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a80848b-7423-4904-83dd-fec0d859a7db" containerName="revoke-keys"
Sep 11 22:11:02.304230 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.304190    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a80848b-7423-4904-83dd-fec0d859a7db" containerName="revoke-keys"
Sep 11 22:11:02.307224 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.307206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:02.309459 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.309440    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-92d3298e-dockercfg-nxwtr\""
Sep 11 22:11:02.309554 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.309457    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-92d3298e\""
Sep 11 22:11:02.318516 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.318483    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"]
Sep 11 22:11:02.414759 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.414693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htdnn\" (UniqueName: \"kubernetes.io/projected/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-kube-api-access-htdnn\") pod \"payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c\" (UID: \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:02.414953 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.414818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c\" (UID: \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:02.515577 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.515533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c\" (UID: \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:02.515843 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.515813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htdnn\" (UniqueName: \"kubernetes.io/projected/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-kube-api-access-htdnn\") pod \"payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c\" (UID: \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:02.516493 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.516466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c\" (UID: \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:02.526032 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.526000    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"]
Sep 11 22:11:02.526277 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.526231    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj" podUID="0f25974d-fb78-4609-8d2e-acf7eb7d6bca" containerName="payload-processing" containerID="cri-o://0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b" gracePeriod=30
Sep 11 22:11:02.530812 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.530782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htdnn\" (UniqueName: \"kubernetes.io/projected/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-kube-api-access-htdnn\") pod \"payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c\" (UID: \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:02.535136 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.535106    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"]
Sep 11 22:11:02.535379 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.535332    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq" podUID="f825b3ee-7f54-4421-932c-edcaf1d0ff00" containerName="payload-pre-processing" containerID="cri-o://544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88" gracePeriod=30
Sep 11 22:11:02.618608 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.618579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:02.757693 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.757662    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"]
Sep 11 22:11:02.764845 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.764813    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-37182a-mq52g"]
Sep 11 22:11:02.822774 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.822740    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:11:02.825885 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.825860    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:11:02.859820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.859784    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a80848b-7423-4904-83dd-fec0d859a7db" path="/var/lib/kubelet/pods/5a80848b-7423-4904-83dd-fec0d859a7db/volumes"
Sep 11 22:11:02.897129 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.897101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"]
Sep 11 22:11:02.898531 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:11:02.898498    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd4d2fa_48bd_40a2_bd76_7246bb7677d8.slice/crio-7b9cbfd768b571fa0dfb32b7b54b175065fbf80ada070a248f20df86235ead43 WatchSource:0}: Error finding container 7b9cbfd768b571fa0dfb32b7b54b175065fbf80ada070a248f20df86235ead43: Status 404 returned error can't find the container with id 7b9cbfd768b571fa0dfb32b7b54b175065fbf80ada070a248f20df86235ead43
Sep 11 22:11:02.918923 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.918899    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-plugins-config-volume\") pod \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\" (UID: \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\") "
Sep 11 22:11:02.919042 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.918977    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmdbz\" (UniqueName: \"kubernetes.io/projected/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-kube-api-access-wmdbz\") pod \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\" (UID: \"0f25974d-fb78-4609-8d2e-acf7eb7d6bca\") "
Sep 11 22:11:02.919111 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.919069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvvg5\" (UniqueName: \"kubernetes.io/projected/f825b3ee-7f54-4421-932c-edcaf1d0ff00-kube-api-access-nvvg5\") pod \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\" (UID: \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\") "
Sep 11 22:11:02.919167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.919147    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f825b3ee-7f54-4421-932c-edcaf1d0ff00-plugins-config-volume\") pod \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\" (UID: \"f825b3ee-7f54-4421-932c-edcaf1d0ff00\") "
Sep 11 22:11:02.919302 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.919280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f25974d-fb78-4609-8d2e-acf7eb7d6bca" (UID: "0f25974d-fb78-4609-8d2e-acf7eb7d6bca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:11:02.919611 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.919585    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f825b3ee-7f54-4421-932c-edcaf1d0ff00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f825b3ee-7f54-4421-932c-edcaf1d0ff00" (UID: "f825b3ee-7f54-4421-932c-edcaf1d0ff00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:11:02.919725 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.919655    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:02.921103 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.921074    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-kube-api-access-wmdbz" (OuterVolumeSpecName: "kube-api-access-wmdbz") pod "0f25974d-fb78-4609-8d2e-acf7eb7d6bca" (UID: "0f25974d-fb78-4609-8d2e-acf7eb7d6bca"). InnerVolumeSpecName "kube-api-access-wmdbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:02.921103 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:02.921081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f825b3ee-7f54-4421-932c-edcaf1d0ff00-kube-api-access-nvvg5" (OuterVolumeSpecName: "kube-api-access-nvvg5") pod "f825b3ee-7f54-4421-932c-edcaf1d0ff00" (UID: "f825b3ee-7f54-4421-932c-edcaf1d0ff00"). InnerVolumeSpecName "kube-api-access-nvvg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:03.020967 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.020936    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmdbz\" (UniqueName: \"kubernetes.io/projected/0f25974d-fb78-4609-8d2e-acf7eb7d6bca-kube-api-access-wmdbz\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:03.020967 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.020964    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvg5\" (UniqueName: \"kubernetes.io/projected/f825b3ee-7f54-4421-932c-edcaf1d0ff00-kube-api-access-nvvg5\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:03.020967 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.020975    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f825b3ee-7f54-4421-932c-edcaf1d0ff00-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:03.025282 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.025245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c" event={"ID":"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8","Type":"ContainerStarted","Data":"ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61"}
Sep 11 22:11:03.025445 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.025311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c" event={"ID":"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8","Type":"ContainerStarted","Data":"7b9cbfd768b571fa0dfb32b7b54b175065fbf80ada070a248f20df86235ead43"}
Sep 11 22:11:03.025506 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.025475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:03.026998 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.026966    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f25974d-fb78-4609-8d2e-acf7eb7d6bca" containerID="0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b" exitCode=0
Sep 11 22:11:03.027131 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.027032    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"
Sep 11 22:11:03.027131 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.027058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj" event={"ID":"0f25974d-fb78-4609-8d2e-acf7eb7d6bca","Type":"ContainerDied","Data":"0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b"}
Sep 11 22:11:03.027131 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.027116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj" event={"ID":"0f25974d-fb78-4609-8d2e-acf7eb7d6bca","Type":"ContainerDied","Data":"e880bd13238d14b3545c1affcfb9fc418305c240fce2b1560f33155331dda628"}
Sep 11 22:11:03.027310 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.027146    2578 scope.go:117] "RemoveContainer" containerID="0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b"
Sep 11 22:11:03.029103 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.029078    2578 generic.go:356] "Generic (PLEG): container finished" podID="f825b3ee-7f54-4421-932c-edcaf1d0ff00" containerID="544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88" exitCode=0
Sep 11 22:11:03.029199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.029113    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"
Sep 11 22:11:03.029199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.029118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq" event={"ID":"f825b3ee-7f54-4421-932c-edcaf1d0ff00","Type":"ContainerDied","Data":"544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88"}
Sep 11 22:11:03.029199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.029162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq" event={"ID":"f825b3ee-7f54-4421-932c-edcaf1d0ff00","Type":"ContainerDied","Data":"97521eb70b85cd2029abda63b0fa25cbc3ca1ab045196fe5145aa5a02fe14874"}
Sep 11 22:11:03.038900 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.038875    2578 scope.go:117] "RemoveContainer" containerID="0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b"
Sep 11 22:11:03.039313 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:11:03.039273    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b\": container with ID starting with 0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b not found: ID does not exist" containerID="0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b"
Sep 11 22:11:03.039434 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.039313    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b"} err="failed to get container status \"0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b\": rpc error: code = NotFound desc = could not find container \"0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b\": container with ID starting with 0029a89eba46c8aeb57009a0034d3b8a5637bd55ad2520f563347f6f6142ac7b not found: ID does not exist"
Sep 11 22:11:03.039434 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.039342    2578 scope.go:117] "RemoveContainer" containerID="544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88"
Sep 11 22:11:03.042602 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.042550    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c" podStartSLOduration=1.042533214 podStartE2EDuration="1.042533214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:03.040192423 +0000 UTC m=+1124.725139139" watchObservedRunningTime="2026-09-11 22:11:03.042533214 +0000 UTC m=+1124.727479931"
Sep 11 22:11:03.049114 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.049092    2578 scope.go:117] "RemoveContainer" containerID="544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88"
Sep 11 22:11:03.049423 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:11:03.049404    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88\": container with ID starting with 544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88 not found: ID does not exist" containerID="544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88"
Sep 11 22:11:03.049485 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.049440    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88"} err="failed to get container status \"544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88\": rpc error: code = NotFound desc = could not find container \"544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88\": container with ID starting with 544d896e56ea9e861a9e30ba8c496be27e3d28238cf8cb87944e75a220d07b88 not found: ID does not exist"
Sep 11 22:11:03.055283 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.055260    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"]
Sep 11 22:11:03.061829 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.061778    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37182a-78f888b798-v7mjq"]
Sep 11 22:11:03.076677 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.076651    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"]
Sep 11 22:11:03.081458 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:03.081438    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37182a-c7cf85cc9-97cnj"]
Sep 11 22:11:04.856889 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:04.856856    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f25974d-fb78-4609-8d2e-acf7eb7d6bca" path="/var/lib/kubelet/pods/0f25974d-fb78-4609-8d2e-acf7eb7d6bca/volumes"
Sep 11 22:11:04.857284 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:04.857220    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f825b3ee-7f54-4421-932c-edcaf1d0ff00" path="/var/lib/kubelet/pods/f825b3ee-7f54-4421-932c-edcaf1d0ff00/volumes"
Sep 11 22:11:08.093724 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:08.093691    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"]
Sep 11 22:11:14.039454 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:14.039422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:11:23.113017 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.112969    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg" podUID="bbf4ab82-6018-4491-ad27-1f5bf057dc7d" containerName="main" containerID="cri-o://9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499" gracePeriod=45
Sep 11 22:11:23.357902 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.357870    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:11:23.425335 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425311    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tmp-dir\") pod \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") "
Sep 11 22:11:23.425495 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tls-certs\") pod \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") "
Sep 11 22:11:23.425495 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-home\") pod \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") "
Sep 11 22:11:23.425495 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425464    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-model-cache\") pod \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") "
Sep 11 22:11:23.425495 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-dshm\") pod \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") "
Sep 11 22:11:23.425733 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425538    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g244d\" (UniqueName: \"kubernetes.io/projected/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-kube-api-access-g244d\") pod \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\" (UID: \"bbf4ab82-6018-4491-ad27-1f5bf057dc7d\") "
Sep 11 22:11:23.425733 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425667    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bbf4ab82-6018-4491-ad27-1f5bf057dc7d" (UID: "bbf4ab82-6018-4491-ad27-1f5bf057dc7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:23.425733 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-home" (OuterVolumeSpecName: "home") pod "bbf4ab82-6018-4491-ad27-1f5bf057dc7d" (UID: "bbf4ab82-6018-4491-ad27-1f5bf057dc7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:23.425901 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "bbf4ab82-6018-4491-ad27-1f5bf057dc7d" (UID: "bbf4ab82-6018-4491-ad27-1f5bf057dc7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:23.425968 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425919    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:23.425968 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425960    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:23.426069 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.425974    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:23.427928 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.427899    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-kube-api-access-g244d" (OuterVolumeSpecName: "kube-api-access-g244d") pod "bbf4ab82-6018-4491-ad27-1f5bf057dc7d" (UID: "bbf4ab82-6018-4491-ad27-1f5bf057dc7d"). InnerVolumeSpecName "kube-api-access-g244d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:23.428044 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.428019    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bbf4ab82-6018-4491-ad27-1f5bf057dc7d" (UID: "bbf4ab82-6018-4491-ad27-1f5bf057dc7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:11:23.428044 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.428022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-dshm" (OuterVolumeSpecName: "dshm") pod "bbf4ab82-6018-4491-ad27-1f5bf057dc7d" (UID: "bbf4ab82-6018-4491-ad27-1f5bf057dc7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:23.527122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.527077    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:23.527122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.527115    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:23.527122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:23.527125    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g244d\" (UniqueName: \"kubernetes.io/projected/bbf4ab82-6018-4491-ad27-1f5bf057dc7d-kube-api-access-g244d\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:24.153255 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.153218    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbf4ab82-6018-4491-ad27-1f5bf057dc7d" containerID="9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499" exitCode=0
Sep 11 22:11:24.153673 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.153263    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"
Sep 11 22:11:24.153673 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.153319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg" event={"ID":"bbf4ab82-6018-4491-ad27-1f5bf057dc7d","Type":"ContainerDied","Data":"9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499"}
Sep 11 22:11:24.153673 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.153369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg" event={"ID":"bbf4ab82-6018-4491-ad27-1f5bf057dc7d","Type":"ContainerDied","Data":"5d9f9c5bf0de28657c2398fa1d7aa02998154c7f1094db811fbb5741368cf75d"}
Sep 11 22:11:24.153673 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.153386    2578 scope.go:117] "RemoveContainer" containerID="9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499"
Sep 11 22:11:24.163230 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.163206    2578 scope.go:117] "RemoveContainer" containerID="9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499"
Sep 11 22:11:24.163506 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:11:24.163489    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499\": container with ID starting with 9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499 not found: ID does not exist" containerID="9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499"
Sep 11 22:11:24.163555 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.163513    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499"} err="failed to get container status \"9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499\": rpc error: code = NotFound desc = could not find container \"9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499\": container with ID starting with 9fbdf91c500c1b8ac84d8afa1c4e360e8aee0780ba22bfeb750fc7b913997499 not found: ID does not exist"
Sep 11 22:11:24.174254 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.174222    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"]
Sep 11 22:11:24.177216 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.177189    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-37182a/test-model-37182a-kserve-778cd96cd9-jf7hg"]
Sep 11 22:11:24.860183 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:24.860145    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf4ab82-6018-4491-ad27-1f5bf057dc7d" path="/var/lib/kubelet/pods/bbf4ab82-6018-4491-ad27-1f5bf057dc7d/volumes"
Sep 11 22:11:27.795201 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795167    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"]
Sep 11 22:11:27.795604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795553    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f25974d-fb78-4609-8d2e-acf7eb7d6bca" containerName="payload-processing"
Sep 11 22:11:27.795604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795564    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f25974d-fb78-4609-8d2e-acf7eb7d6bca" containerName="payload-processing"
Sep 11 22:11:27.795604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795577    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf4ab82-6018-4491-ad27-1f5bf057dc7d" containerName="main"
Sep 11 22:11:27.795604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795583    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf4ab82-6018-4491-ad27-1f5bf057dc7d" containerName="main"
Sep 11 22:11:27.795604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795599    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f825b3ee-7f54-4421-932c-edcaf1d0ff00" containerName="payload-pre-processing"
Sep 11 22:11:27.795604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795606    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f825b3ee-7f54-4421-932c-edcaf1d0ff00" containerName="payload-pre-processing"
Sep 11 22:11:27.795843 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795689    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f825b3ee-7f54-4421-932c-edcaf1d0ff00" containerName="payload-pre-processing"
Sep 11 22:11:27.795843 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795700    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f25974d-fb78-4609-8d2e-acf7eb7d6bca" containerName="payload-processing"
Sep 11 22:11:27.795843 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.795707    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf4ab82-6018-4491-ad27-1f5bf057dc7d" containerName="main"
Sep 11 22:11:27.800011 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.799990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.803218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.802898    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-26d968\"/\"kube-root-ca.crt\""
Sep 11 22:11:27.803218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.803017    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-26d968\"/\"openshift-service-ca.crt\""
Sep 11 22:11:27.803218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.803161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-26d968\"/\"default-dockercfg-g6wtp\""
Sep 11 22:11:27.803466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.803315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-26d968\"/\"noten-model-26d968-kserve-self-signed-certs\""
Sep 11 22:11:27.805928 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.805904    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"]
Sep 11 22:11:27.871602 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.871567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-dshm\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.871803 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.871655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tls-certs\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.871803 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.871683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-home\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.871803 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.871703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldpf5\" (UniqueName: \"kubernetes.io/projected/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-kube-api-access-ldpf5\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.871803 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.871755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-model-cache\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.871803 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.871772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tmp-dir\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.972965 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.972924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-dshm\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.973151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.972990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tls-certs\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.973151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.973011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-home\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.973151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.973030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpf5\" (UniqueName: \"kubernetes.io/projected/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-kube-api-access-ldpf5\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.973151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.973085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-model-cache\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.973151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.973108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tmp-dir\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.973575 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.973514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-model-cache\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.973779 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.973698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tmp-dir\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.973993 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.973971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-home\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.975235 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.975214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-dshm\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.975483 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.975466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tls-certs\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:27.982576 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:27.982550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldpf5\" (UniqueName: \"kubernetes.io/projected/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-kube-api-access-ldpf5\") pod \"noten-model-26d968-kserve-7ddd8cb89f-xt4db\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") " pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:28.110932 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:28.110841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:28.382355 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:28.382324    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"]
Sep 11 22:11:28.383725 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:11:28.383692    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ff2ecaa_364e_49b3_a758_0e04fbf6591a.slice/crio-5d3b9f27e40bb5fe2fd3b3cac9628c6f4d92cd9e70ca13a50c4f0f126907ccf6 WatchSource:0}: Error finding container 5d3b9f27e40bb5fe2fd3b3cac9628c6f4d92cd9e70ca13a50c4f0f126907ccf6: Status 404 returned error can't find the container with id 5d3b9f27e40bb5fe2fd3b3cac9628c6f4d92cd9e70ca13a50c4f0f126907ccf6
Sep 11 22:11:29.186705 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:29.186668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db" event={"ID":"4ff2ecaa-364e-49b3-a758-0e04fbf6591a","Type":"ContainerStarted","Data":"bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa"}
Sep 11 22:11:29.187091 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:29.186715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db" event={"ID":"4ff2ecaa-364e-49b3-a758-0e04fbf6591a","Type":"ContainerStarted","Data":"5d3b9f27e40bb5fe2fd3b3cac9628c6f4d92cd9e70ca13a50c4f0f126907ccf6"}
Sep 11 22:11:29.202624 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:29.202575    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db" podStartSLOduration=2.202558346 podStartE2EDuration="2.202558346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:29.20124042 +0000 UTC m=+1150.886187135" watchObservedRunningTime="2026-09-11 22:11:29.202558346 +0000 UTC m=+1150.887505105"
Sep 11 22:11:38.111287 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:38.111235    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:38.111811 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:38.111311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:38.124209 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:38.124182    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:38.250397 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:38.250368    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:11:44.977125 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:44.977044    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"]
Sep 11 22:11:44.977530 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:44.977329    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy" containerID="cri-o://1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c" gracePeriod=30
Sep 11 22:11:46.325088 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:46.325047    2578 patch_prober.go:28] interesting pod/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:11:46.325528 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:46.325113    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:11:50.231620 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.231584    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:11:50.309662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.309558    2578 generic.go:356] "Generic (PLEG): container finished" podID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerID="1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c" exitCode=0
Sep 11 22:11:50.309662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.309606    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"
Sep 11 22:11:50.309849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.309612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" event={"ID":"852388bd-f055-4887-95b3-9c83b2d4bb83","Type":"ContainerDied","Data":"1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c"}
Sep 11 22:11:50.309849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.309792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th" event={"ID":"852388bd-f055-4887-95b3-9c83b2d4bb83","Type":"ContainerDied","Data":"b2cb7b2f082a668392b99d957a1b22123a02fd7687ab19e0743aba7c297261b5"}
Sep 11 22:11:50.309849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.309833    2578 scope.go:117] "RemoveContainer" containerID="1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c"
Sep 11 22:11:50.319280 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.319261    2578 scope.go:117] "RemoveContainer" containerID="fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a"
Sep 11 22:11:50.328272 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.328250    2578 scope.go:117] "RemoveContainer" containerID="1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c"
Sep 11 22:11:50.328598 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:11:50.328576    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c\": container with ID starting with 1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c not found: ID does not exist" containerID="1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c"
Sep 11 22:11:50.328719 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.328608    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c"} err="failed to get container status \"1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c\": rpc error: code = NotFound desc = could not find container \"1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c\": container with ID starting with 1deb6540e34bb7ae2d8ae6a6e7451c73758e251302bcaa942371f13a06867a7c not found: ID does not exist"
Sep 11 22:11:50.328719 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.328627    2578 scope.go:117] "RemoveContainer" containerID="fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a"
Sep 11 22:11:50.328924 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:11:50.328902    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a\": container with ID starting with fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a not found: ID does not exist" containerID="fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a"
Sep 11 22:11:50.328973 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.328930    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a"} err="failed to get container status \"fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a\": rpc error: code = NotFound desc = could not find container \"fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a\": container with ID starting with fe3f863d2137fbb6e6814a5e3bc90b852e57f89a03592ced7542556dbbf1911a not found: ID does not exist"
Sep 11 22:11:50.394017 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.393986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-data\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-envoy\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-credential-socket\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394105    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852388bd-f055-4887-95b3-9c83b2d4bb83-istiod-ca-cert\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394133    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-certs\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfrv8\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-kube-api-access-xfrv8\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394451 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-podinfo\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394451 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-token\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394451 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-socket\") pod \"852388bd-f055-4887-95b3-9c83b2d4bb83\" (UID: \"852388bd-f055-4887-95b3-9c83b2d4bb83\") "
Sep 11 22:11:50.394451 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394399    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:50.394451 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394297    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-data" (OuterVolumeSpecName: "istio-data") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:50.394779 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852388bd-f055-4887-95b3-9c83b2d4bb83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:11:50.394779 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:50.394779 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394682    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:50.394941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394839    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.394941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394855    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852388bd-f055-4887-95b3-9c83b2d4bb83-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.394941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394864    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.394941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394876    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.394941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.394888    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.396389 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.396366    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:50.396496 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.396437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-kube-api-access-xfrv8" (OuterVolumeSpecName: "kube-api-access-xfrv8") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "kube-api-access-xfrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:50.396821 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.396795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:11:50.396921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.396795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-token" (OuterVolumeSpecName: "istio-token") pod "852388bd-f055-4887-95b3-9c83b2d4bb83" (UID: "852388bd-f055-4887-95b3-9c83b2d4bb83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:50.495788 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.495746    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.495788 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.495783    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrv8\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-kube-api-access-xfrv8\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.495788 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.495794    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.496036 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.495802    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852388bd-f055-4887-95b3-9c83b2d4bb83-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:11:50.631280 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.631242    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"]
Sep 11 22:11:50.634668 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.634628    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-37182a-openshift-default-77c66888d-qr5th"]
Sep 11 22:11:50.857291 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:50.857254    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" path="/var/lib/kubelet/pods/852388bd-f055-4887-95b3-9c83b2d4bb83/volumes"
Sep 11 22:11:53.110793 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.110747    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"]
Sep 11 22:11:53.111324 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.111302    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy"
Sep 11 22:11:53.111412 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.111328    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy"
Sep 11 22:11:53.111412 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.111340    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy"
Sep 11 22:11:53.111412 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.111348    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy"
Sep 11 22:11:53.111561 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.111450    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy"
Sep 11 22:11:53.111561 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.111464    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="852388bd-f055-4887-95b3-9c83b2d4bb83" containerName="istio-proxy"
Sep 11 22:11:53.116116 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.116095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:53.119039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.119022    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d09c96\""
Sep 11 22:11:53.119478 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.119464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d09c96-dockercfg-6hgrx\""
Sep 11 22:11:53.128923 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.128900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"]
Sep 11 22:11:53.139807 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.139766    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"]
Sep 11 22:11:53.143610 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.143588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:53.153275 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.153246    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"]
Sep 11 22:11:53.224005 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.223968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54933a8e-de7f-44a9-9c55-b21e3ece54a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2\" (UID: \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:53.224214 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.224102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjqnm\" (UniqueName: \"kubernetes.io/projected/54933a8e-de7f-44a9-9c55-b21e3ece54a8-kube-api-access-cjqnm\") pod \"payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2\" (UID: \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:53.325271 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.325229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjqnm\" (UniqueName: \"kubernetes.io/projected/54933a8e-de7f-44a9-9c55-b21e3ece54a8-kube-api-access-cjqnm\") pod \"payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2\" (UID: \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:53.325477 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.325281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30854b6d-6fc1-4d3b-8e27-edef38c380a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw\" (UID: \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:53.325477 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.325385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54933a8e-de7f-44a9-9c55-b21e3ece54a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2\" (UID: \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:53.325477 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.325433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xk5t\" (UniqueName: \"kubernetes.io/projected/30854b6d-6fc1-4d3b-8e27-edef38c380a7-kube-api-access-2xk5t\") pod \"payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw\" (UID: \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:53.326081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.326061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54933a8e-de7f-44a9-9c55-b21e3ece54a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2\" (UID: \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:53.338862 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.338834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjqnm\" (UniqueName: \"kubernetes.io/projected/54933a8e-de7f-44a9-9c55-b21e3ece54a8-kube-api-access-cjqnm\") pod \"payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2\" (UID: \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:53.426409 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.426370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:53.426620 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.426595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30854b6d-6fc1-4d3b-8e27-edef38c380a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw\" (UID: \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:53.426730 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.426716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xk5t\" (UniqueName: \"kubernetes.io/projected/30854b6d-6fc1-4d3b-8e27-edef38c380a7-kube-api-access-2xk5t\") pod \"payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw\" (UID: \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:53.427282 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.427263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30854b6d-6fc1-4d3b-8e27-edef38c380a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw\" (UID: \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:53.440063 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.440015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xk5t\" (UniqueName: \"kubernetes.io/projected/30854b6d-6fc1-4d3b-8e27-edef38c380a7-kube-api-access-2xk5t\") pod \"payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw\" (UID: \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:53.454807 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.454769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:53.703078 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.703048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"]
Sep 11 22:11:53.704660 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:11:53.704622    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54933a8e_de7f_44a9_9c55_b21e3ece54a8.slice/crio-330f1f750424c553683db00b6fe0f52e111c945e6865e7975312320431d92c49 WatchSource:0}: Error finding container 330f1f750424c553683db00b6fe0f52e111c945e6865e7975312320431d92c49: Status 404 returned error can't find the container with id 330f1f750424c553683db00b6fe0f52e111c945e6865e7975312320431d92c49
Sep 11 22:11:53.734448 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:53.734422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"]
Sep 11 22:11:53.744894 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:11:53.744866    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30854b6d_6fc1_4d3b_8e27_edef38c380a7.slice/crio-a826195a942ddc6cd46d318120689d52be8f49101b8360020de887c98d76e426 WatchSource:0}: Error finding container a826195a942ddc6cd46d318120689d52be8f49101b8360020de887c98d76e426: Status 404 returned error can't find the container with id a826195a942ddc6cd46d318120689d52be8f49101b8360020de887c98d76e426
Sep 11 22:11:54.338232 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:54.338193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2" event={"ID":"54933a8e-de7f-44a9-9c55-b21e3ece54a8","Type":"ContainerStarted","Data":"f2a56967c68d3f7adc460417866107ccf098797b758e90600296ecedc0ddf81c"}
Sep 11 22:11:54.338740 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:54.338260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2" event={"ID":"54933a8e-de7f-44a9-9c55-b21e3ece54a8","Type":"ContainerStarted","Data":"330f1f750424c553683db00b6fe0f52e111c945e6865e7975312320431d92c49"}
Sep 11 22:11:54.338740 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:54.338292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:11:54.340164 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:54.340137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw" event={"ID":"30854b6d-6fc1-4d3b-8e27-edef38c380a7","Type":"ContainerStarted","Data":"1ee703290fd4e76570095b80a99560b1e233f4ee0f6aa14e3224a8e0abd95023"}
Sep 11 22:11:54.340305 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:54.340187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw" event={"ID":"30854b6d-6fc1-4d3b-8e27-edef38c380a7","Type":"ContainerStarted","Data":"a826195a942ddc6cd46d318120689d52be8f49101b8360020de887c98d76e426"}
Sep 11 22:11:54.340305 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:54.340218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:11:54.355407 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:54.355355    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2" podStartSLOduration=1.355337751 podStartE2EDuration="1.355337751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:54.353780787 +0000 UTC m=+1176.038727502" watchObservedRunningTime="2026-09-11 22:11:54.355337751 +0000 UTC m=+1176.040284470"
Sep 11 22:11:54.369781 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:54.369726    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw" podStartSLOduration=1.369709887 podStartE2EDuration="1.369709887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:54.369503525 +0000 UTC m=+1176.054450240" watchObservedRunningTime="2026-09-11 22:11:54.369709887 +0000 UTC m=+1176.054656602"
Sep 11 22:11:55.762783 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.762744    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"]
Sep 11 22:11:55.766454 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.766432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:55.768309 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.768289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"e2e-conflict-b-f25d30a7-kserve-self-signed-certs\""
Sep 11 22:11:55.775046 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.775022    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"]
Sep 11 22:11:55.951045 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.951004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-model-cache\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:55.951045 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.951047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-tmp-dir\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:55.951268 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.951119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-dshm\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:55.951268 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.951239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttccz\" (UniqueName: \"kubernetes.io/projected/63a36b2e-3feb-45bb-bbb0-45eed2987393-kube-api-access-ttccz\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:55.951350 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.951276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a36b2e-3feb-45bb-bbb0-45eed2987393-tls-certs\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:55.951393 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:55.951370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-home\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.052702 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.052598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-home\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.052888 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.052709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-model-cache\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.052888 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.052743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-tmp-dir\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.052888 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.052873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-dshm\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.053065 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.052982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttccz\" (UniqueName: \"kubernetes.io/projected/63a36b2e-3feb-45bb-bbb0-45eed2987393-kube-api-access-ttccz\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.053065 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.053019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a36b2e-3feb-45bb-bbb0-45eed2987393-tls-certs\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.053065 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.053041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-home\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.053213 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.053117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-tmp-dir\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.053213 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.053125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-model-cache\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.060680 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.055348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-dshm\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.060680 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.055822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a36b2e-3feb-45bb-bbb0-45eed2987393-tls-certs\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.062535 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.062504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttccz\" (UniqueName: \"kubernetes.io/projected/63a36b2e-3feb-45bb-bbb0-45eed2987393-kube-api-access-ttccz\") pod \"e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") " pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.077024 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.076992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:11:56.353909 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:56.353870    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"]
Sep 11 22:11:56.355035 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:11:56.355011    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63a36b2e_3feb_45bb_bbb0_45eed2987393.slice/crio-3136f96c832f43235fc0d0542623e640f880b164ca3701d78da356e6484ac2a6 WatchSource:0}: Error finding container 3136f96c832f43235fc0d0542623e640f880b164ca3701d78da356e6484ac2a6: Status 404 returned error can't find the container with id 3136f96c832f43235fc0d0542623e640f880b164ca3701d78da356e6484ac2a6
Sep 11 22:11:57.363374 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:57.363333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x" event={"ID":"63a36b2e-3feb-45bb-bbb0-45eed2987393","Type":"ContainerStarted","Data":"6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f"}
Sep 11 22:11:57.363374 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:57.363380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x" event={"ID":"63a36b2e-3feb-45bb-bbb0-45eed2987393","Type":"ContainerStarted","Data":"3136f96c832f43235fc0d0542623e640f880b164ca3701d78da356e6484ac2a6"}
Sep 11 22:11:57.409207 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:11:57.409146    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x" podStartSLOduration=2.40912968 podStartE2EDuration="2.40912968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:57.406720284 +0000 UTC m=+1179.091667001" watchObservedRunningTime="2026-09-11 22:11:57.40912968 +0000 UTC m=+1179.094076394"
Sep 11 22:12:05.348834 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:05.348797    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:12:05.349258 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:05.349070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:12:06.077946 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:06.077909    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:12:06.078138 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:06.077963    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:12:06.090251 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:06.090217    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:12:06.427897 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:06.427866    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:12:06.858041 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:06.857961    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"]
Sep 11 22:12:18.938411 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:18.938369    2578 scope.go:117] "RemoveContainer" containerID="3b1be81ba9c0137cd1ae4f454ea0ee0c03518feedd235e75d59db3ad415489cf"
Sep 11 22:12:19.186403 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.186370    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"]
Sep 11 22:12:19.191203 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.191139    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.193117 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.193096    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-966527-openshift-default-dockercfg-hkkcv\""
Sep 11 22:12:19.200011 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.199985    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"]
Sep 11 22:12:19.285691 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.285655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-socket\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.285857 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.285703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istiod-ca-cert\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.285857 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.285799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-podinfo\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.285857 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.285831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5wt4\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-kube-api-access-n5wt4\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.285857 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.285854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-credential-socket\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.286099 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.285951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-data\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.286099 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.285993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-certs\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.286099 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.286027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-token\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.286099 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.286062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-envoy\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387152 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-envoy\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387331 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-socket\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387331 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istiod-ca-cert\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387331 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-podinfo\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387331 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5wt4\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-kube-api-access-n5wt4\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387331 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-credential-socket\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387572 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-data\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387572 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-certs\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387572 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-token\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387746 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-socket\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.387904 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-data\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.388008 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.387987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istiod-ca-cert\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.388081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.388026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-credential-socket\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.388081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.388066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-certs\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.389459 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.389433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-envoy\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.389876 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.389859    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-podinfo\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.395871 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.395847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-token\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.396557 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.396539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5wt4\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-kube-api-access-n5wt4\") pod \"e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") " pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.503699 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.503578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:19.781568 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.781534    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"]
Sep 11 22:12:19.782931 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:12:19.782901    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d63d2ef_3367_4a52_a706_8ba66a1c17c6.slice/crio-a8ec017252001b7cc75c78140a28e03db004ea9463b188e6f7fd6a01937c3d82 WatchSource:0}: Error finding container a8ec017252001b7cc75c78140a28e03db004ea9463b188e6f7fd6a01937c3d82: Status 404 returned error can't find the container with id a8ec017252001b7cc75c78140a28e03db004ea9463b188e6f7fd6a01937c3d82
Sep 11 22:12:19.785474 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.785433    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:12:19.785552 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.785536    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:12:19.785604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:19.785572    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:12:20.506098 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:20.506060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" event={"ID":"9d63d2ef-3367-4a52-a706-8ba66a1c17c6","Type":"ContainerStarted","Data":"a93ba515074d44d88fbc611d1ba88e5c885c7e672da904ac1c8207b1c79f36e7"}
Sep 11 22:12:20.506503 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:20.506114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" event={"ID":"9d63d2ef-3367-4a52-a706-8ba66a1c17c6","Type":"ContainerStarted","Data":"a8ec017252001b7cc75c78140a28e03db004ea9463b188e6f7fd6a01937c3d82"}
Sep 11 22:12:20.524415 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:20.524350    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" podStartSLOduration=1.5242981370000002 podStartE2EDuration="1.524298137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:12:20.52350787 +0000 UTC m=+1202.208454598" watchObservedRunningTime="2026-09-11 22:12:20.524298137 +0000 UTC m=+1202.209244852"
Sep 11 22:12:21.504695 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:21.504659    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:21.510449 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:21.510419    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:21.512025 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:21.512008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:21.512670 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:21.512629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:21.885219 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:21.885123    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db" podUID="4ff2ecaa-364e-49b3-a758-0e04fbf6591a" containerName="main" containerID="cri-o://bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa" gracePeriod=45
Sep 11 22:12:22.149366 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.149307    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:12:22.317303 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tls-certs\") pod \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") "
Sep 11 22:12:22.317497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-home\") pod \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") "
Sep 11 22:12:22.317497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317420    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-model-cache\") pod \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") "
Sep 11 22:12:22.317497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldpf5\" (UniqueName: \"kubernetes.io/projected/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-kube-api-access-ldpf5\") pod \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") "
Sep 11 22:12:22.317497 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tmp-dir\") pod \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") "
Sep 11 22:12:22.317773 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317540    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-dshm\") pod \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\" (UID: \"4ff2ecaa-364e-49b3-a758-0e04fbf6591a\") "
Sep 11 22:12:22.317773 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317685    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ff2ecaa-364e-49b3-a758-0e04fbf6591a" (UID: "4ff2ecaa-364e-49b3-a758-0e04fbf6591a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:22.317773 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-home" (OuterVolumeSpecName: "home") pod "4ff2ecaa-364e-49b3-a758-0e04fbf6591a" (UID: "4ff2ecaa-364e-49b3-a758-0e04fbf6591a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:22.317926 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ff2ecaa-364e-49b3-a758-0e04fbf6591a" (UID: "4ff2ecaa-364e-49b3-a758-0e04fbf6591a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:22.317992 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317979    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:12:22.318059 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.317994    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:12:22.318059 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.318007    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:12:22.319750 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.319721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-kube-api-access-ldpf5" (OuterVolumeSpecName: "kube-api-access-ldpf5") pod "4ff2ecaa-364e-49b3-a758-0e04fbf6591a" (UID: "4ff2ecaa-364e-49b3-a758-0e04fbf6591a"). InnerVolumeSpecName "kube-api-access-ldpf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:12:22.319849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.319758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-dshm" (OuterVolumeSpecName: "dshm") pod "4ff2ecaa-364e-49b3-a758-0e04fbf6591a" (UID: "4ff2ecaa-364e-49b3-a758-0e04fbf6591a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:22.319849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.319759    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ff2ecaa-364e-49b3-a758-0e04fbf6591a" (UID: "4ff2ecaa-364e-49b3-a758-0e04fbf6591a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:12:22.419444 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.419418    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldpf5\" (UniqueName: \"kubernetes.io/projected/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-kube-api-access-ldpf5\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:12:22.419603 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.419447    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:12:22.419603 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.419458    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ff2ecaa-364e-49b3-a758-0e04fbf6591a-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:12:22.518433 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.518396    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ff2ecaa-364e-49b3-a758-0e04fbf6591a" containerID="bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa" exitCode=0
Sep 11 22:12:22.518869 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.518442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"
Sep 11 22:12:22.518869 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.518440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db" event={"ID":"4ff2ecaa-364e-49b3-a758-0e04fbf6591a","Type":"ContainerDied","Data":"bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa"}
Sep 11 22:12:22.518869 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.518497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db" event={"ID":"4ff2ecaa-364e-49b3-a758-0e04fbf6591a","Type":"ContainerDied","Data":"5d3b9f27e40bb5fe2fd3b3cac9628c6f4d92cd9e70ca13a50c4f0f126907ccf6"}
Sep 11 22:12:22.518869 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.518517    2578 scope.go:117] "RemoveContainer" containerID="bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa"
Sep 11 22:12:22.528626 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.528609    2578 scope.go:117] "RemoveContainer" containerID="bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa"
Sep 11 22:12:22.528946 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:12:22.528923    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa\": container with ID starting with bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa not found: ID does not exist" containerID="bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa"
Sep 11 22:12:22.528999 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.528955    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa"} err="failed to get container status \"bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa\": rpc error: code = NotFound desc = could not find container \"bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa\": container with ID starting with bb1ebf891c42fe40fffde16d04d0d61f6049138c8048156b5d75079fd94a8eaa not found: ID does not exist"
Sep 11 22:12:22.544296 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.544269    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"]
Sep 11 22:12:22.548493 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.548468    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-26d968/noten-model-26d968-kserve-7ddd8cb89f-xt4db"]
Sep 11 22:12:22.858429 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:22.858345    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ff2ecaa-364e-49b3-a758-0e04fbf6591a" path="/var/lib/kubelet/pods/4ff2ecaa-364e-49b3-a758-0e04fbf6591a/volumes"
Sep 11 22:12:27.808974 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.808935    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"]
Sep 11 22:12:27.809362 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.809345    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff2ecaa-364e-49b3-a758-0e04fbf6591a" containerName="main"
Sep 11 22:12:27.809362 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.809356    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff2ecaa-364e-49b3-a758-0e04fbf6591a" containerName="main"
Sep 11 22:12:27.809439 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.809425    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff2ecaa-364e-49b3-a758-0e04fbf6591a" containerName="main"
Sep 11 22:12:27.814018 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.814001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:27.818037 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.818013    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-966527-dockercfg-9w6cc\""
Sep 11 22:12:27.818037 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.818028    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-966527\""
Sep 11 22:12:27.829324 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.829297    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"]
Sep 11 22:12:27.845595 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.845563    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"]
Sep 11 22:12:27.849057 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.849035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:27.859335 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.859301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"]
Sep 11 22:12:27.975804 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.975768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62dr\" (UniqueName: \"kubernetes.io/projected/ff4770dc-eb19-4b83-854c-93a4603616b7-kube-api-access-n62dr\") pod \"payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp\" (UID: \"ff4770dc-eb19-4b83-854c-93a4603616b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:27.975998 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.975850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ec7970-dab7-4f5e-97d0-556fd6bcad14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9\" (UID: \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:27.975998 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.975891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlcxd\" (UniqueName: \"kubernetes.io/projected/33ec7970-dab7-4f5e-97d0-556fd6bcad14-kube-api-access-hlcxd\") pod \"payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9\" (UID: \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:27.975998 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:27.975969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4770dc-eb19-4b83-854c-93a4603616b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp\" (UID: \"ff4770dc-eb19-4b83-854c-93a4603616b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:28.076876 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.076792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlcxd\" (UniqueName: \"kubernetes.io/projected/33ec7970-dab7-4f5e-97d0-556fd6bcad14-kube-api-access-hlcxd\") pod \"payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9\" (UID: \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:28.076876 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.076871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4770dc-eb19-4b83-854c-93a4603616b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp\" (UID: \"ff4770dc-eb19-4b83-854c-93a4603616b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:28.077090 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.076918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62dr\" (UniqueName: \"kubernetes.io/projected/ff4770dc-eb19-4b83-854c-93a4603616b7-kube-api-access-n62dr\") pod \"payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp\" (UID: \"ff4770dc-eb19-4b83-854c-93a4603616b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:28.077090 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.076976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ec7970-dab7-4f5e-97d0-556fd6bcad14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9\" (UID: \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:28.077726 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.077705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ec7970-dab7-4f5e-97d0-556fd6bcad14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9\" (UID: \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:28.077726 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.077721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4770dc-eb19-4b83-854c-93a4603616b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp\" (UID: \"ff4770dc-eb19-4b83-854c-93a4603616b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:28.085442 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.085414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlcxd\" (UniqueName: \"kubernetes.io/projected/33ec7970-dab7-4f5e-97d0-556fd6bcad14-kube-api-access-hlcxd\") pod \"payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9\" (UID: \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:28.085534 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.085501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62dr\" (UniqueName: \"kubernetes.io/projected/ff4770dc-eb19-4b83-854c-93a4603616b7-kube-api-access-n62dr\") pod \"payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp\" (UID: \"ff4770dc-eb19-4b83-854c-93a4603616b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:28.123547 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.123521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:28.162265 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.161819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:28.395051 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.395026    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"]
Sep 11 22:12:28.395970 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:12:28.395948    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4770dc_eb19_4b83_854c_93a4603616b7.slice/crio-4a5a6cf2f6f13f11d2ff0fb118b4f6aa5abcf72dac559cda0134fd04aee7eb67 WatchSource:0}: Error finding container 4a5a6cf2f6f13f11d2ff0fb118b4f6aa5abcf72dac559cda0134fd04aee7eb67: Status 404 returned error can't find the container with id 4a5a6cf2f6f13f11d2ff0fb118b4f6aa5abcf72dac559cda0134fd04aee7eb67
Sep 11 22:12:28.442496 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.442464    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"]
Sep 11 22:12:28.454853 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:12:28.454824    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ec7970_dab7_4f5e_97d0_556fd6bcad14.slice/crio-9314a1d7de78191d192172f691bbdd4c411c33b81ed3af33433b96acd70dbc7b WatchSource:0}: Error finding container 9314a1d7de78191d192172f691bbdd4c411c33b81ed3af33433b96acd70dbc7b: Status 404 returned error can't find the container with id 9314a1d7de78191d192172f691bbdd4c411c33b81ed3af33433b96acd70dbc7b
Sep 11 22:12:28.558332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.558291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp" event={"ID":"ff4770dc-eb19-4b83-854c-93a4603616b7","Type":"ContainerStarted","Data":"b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3"}
Sep 11 22:12:28.558539 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.558439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:28.558539 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.558470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp" event={"ID":"ff4770dc-eb19-4b83-854c-93a4603616b7","Type":"ContainerStarted","Data":"4a5a6cf2f6f13f11d2ff0fb118b4f6aa5abcf72dac559cda0134fd04aee7eb67"}
Sep 11 22:12:28.560316 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.560289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9" event={"ID":"33ec7970-dab7-4f5e-97d0-556fd6bcad14","Type":"ContainerStarted","Data":"473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c"}
Sep 11 22:12:28.560426 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.560341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9" event={"ID":"33ec7970-dab7-4f5e-97d0-556fd6bcad14","Type":"ContainerStarted","Data":"9314a1d7de78191d192172f691bbdd4c411c33b81ed3af33433b96acd70dbc7b"}
Sep 11 22:12:28.560426 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.560371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:28.575773 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.575717    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp" podStartSLOduration=1.57570034 podStartE2EDuration="1.57570034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:12:28.574015247 +0000 UTC m=+1210.258961973" watchObservedRunningTime="2026-09-11 22:12:28.57570034 +0000 UTC m=+1210.260647053"
Sep 11 22:12:28.589968 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:28.589911    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9" podStartSLOduration=1.5898928570000002 podStartE2EDuration="1.589892857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:12:28.587976008 +0000 UTC m=+1210.272922724" watchObservedRunningTime="2026-09-11 22:12:28.589892857 +0000 UTC m=+1210.274839621"
Sep 11 22:12:39.569267 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:39.569230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:12:39.569683 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:39.569353    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:12:55.721714 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:55.721679    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerID="a93ba515074d44d88fbc611d1ba88e5c885c7e672da904ac1c8207b1c79f36e7" exitCode=0
Sep 11 22:12:55.721714 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:55.721718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" event={"ID":"9d63d2ef-3367-4a52-a706-8ba66a1c17c6","Type":"ContainerDied","Data":"a93ba515074d44d88fbc611d1ba88e5c885c7e672da904ac1c8207b1c79f36e7"}
Sep 11 22:12:55.722171 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:55.722105    2578 scope.go:117] "RemoveContainer" containerID="a93ba515074d44d88fbc611d1ba88e5c885c7e672da904ac1c8207b1c79f36e7"
Sep 11 22:12:55.723295 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:55.723258    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:12:55.723388 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:55.723329    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:12:55.723388 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:55.723359    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 22:12:56.750010 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:56.749966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" event={"ID":"9d63d2ef-3367-4a52-a706-8ba66a1c17c6","Type":"ContainerStarted","Data":"9d933a6879bc4585a2bf62607315e776e629c0d0b812927fdcbd58198128833c"}
Sep 11 22:12:57.504344 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:57.504299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:12:57.504578 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:57.504544    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 22:12:57.504701 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:57.504608    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 22:12:58.504868 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:58.504828    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 22:12:58.505275 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:58.504888    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 22:12:59.504981 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:59.504945    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 22:12:59.505421 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:12:59.505007    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 22:13:00.504662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:00.504602    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 22:13:00.504832 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:00.504686    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 22:13:01.504530 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:01.504488    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 22:13:01.504951 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:01.504551    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 22:13:02.508310 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:02.508272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:13:02.508723 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:02.508539    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:13:02.509137 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:02.509116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:13:04.385692 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.385654    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"]
Sep 11 22:13:04.390332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.390315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.392414 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.392394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-92d3298e\"/\"kube-root-ca.crt\""
Sep 11 22:13:04.392996 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.392967    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-92d3298e\"/\"e2e-lifecycle-model-92d3298e-kserve-self-signed-certs\""
Sep 11 22:13:04.393117 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.392968    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-92d3298e\"/\"openshift-service-ca.crt\""
Sep 11 22:13:04.393305 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.393289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-92d3298e\"/\"default-dockercfg-cpwmd\""
Sep 11 22:13:04.398504 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.398466    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"]
Sep 11 22:13:04.524106 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.524072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-dshm\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.524106 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.524113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-home\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.524338 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.524196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6prj\" (UniqueName: \"kubernetes.io/projected/e4d07f2e-b344-4101-b706-9b35f8f238bc-kube-api-access-l6prj\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.524338 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.524254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-tmp-dir\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.524338 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.524274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-model-cache\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.524338 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.524297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d07f2e-b344-4101-b706-9b35f8f238bc-tls-certs\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.624837 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.624779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-home\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.624990 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.624908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6prj\" (UniqueName: \"kubernetes.io/projected/e4d07f2e-b344-4101-b706-9b35f8f238bc-kube-api-access-l6prj\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.624990 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.624948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-tmp-dir\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.624990 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.624974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-model-cache\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.625169 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.625006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d07f2e-b344-4101-b706-9b35f8f238bc-tls-certs\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.625169 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.625098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-dshm\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.625306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.625284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-home\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.625368 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.625286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-tmp-dir\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.625428 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.625384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-model-cache\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.627426 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.627393    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-dshm\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.627685 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.627668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d07f2e-b344-4101-b706-9b35f8f238bc-tls-certs\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.633866 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.633822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6prj\" (UniqueName: \"kubernetes.io/projected/e4d07f2e-b344-4101-b706-9b35f8f238bc-kube-api-access-l6prj\") pod \"e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") " pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.700941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.700901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:04.972780 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:04.972752    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"]
Sep 11 22:13:04.973962 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:13:04.973930    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4d07f2e_b344_4101_b706_9b35f8f238bc.slice/crio-2b33247f32f288e92d0d3e872693730515b50e50e0d26f5b7de90629c165f349 WatchSource:0}: Error finding container 2b33247f32f288e92d0d3e872693730515b50e50e0d26f5b7de90629c165f349: Status 404 returned error can't find the container with id 2b33247f32f288e92d0d3e872693730515b50e50e0d26f5b7de90629c165f349
Sep 11 22:13:05.802698 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:05.802653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz" event={"ID":"e4d07f2e-b344-4101-b706-9b35f8f238bc","Type":"ContainerStarted","Data":"a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995"}
Sep 11 22:13:05.803138 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:05.802708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz" event={"ID":"e4d07f2e-b344-4101-b706-9b35f8f238bc","Type":"ContainerStarted","Data":"2b33247f32f288e92d0d3e872693730515b50e50e0d26f5b7de90629c165f349"}
Sep 11 22:13:05.817958 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:05.817911    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz" podStartSLOduration=1.817896892 podStartE2EDuration="1.817896892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:13:05.816624197 +0000 UTC m=+1247.501570921" watchObservedRunningTime="2026-09-11 22:13:05.817896892 +0000 UTC m=+1247.502843606"
Sep 11 22:13:13.273282 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.273190    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"]
Sep 11 22:13:13.276907 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.276890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.278991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.278968    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-95d6c2\"/\"distinct-2-w0-95d6c2-kserve-self-signed-certs\""
Sep 11 22:13:13.286812 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.286785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"]
Sep 11 22:13:13.308952 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.308917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-home\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.309120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.308965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-model-cache\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.309120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.308990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-dshm\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.309120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.309052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf2eca8-f317-4e47-9909-e882d5882041-tls-certs\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.309120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.309108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-tmp-dir\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.309276 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.309201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpqb9\" (UniqueName: \"kubernetes.io/projected/caf2eca8-f317-4e47-9909-e882d5882041-kube-api-access-tpqb9\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.410507 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.410461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-tmp-dir\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.410732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.410549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpqb9\" (UniqueName: \"kubernetes.io/projected/caf2eca8-f317-4e47-9909-e882d5882041-kube-api-access-tpqb9\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.410732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.410592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-home\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.410732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.410696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-model-cache\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.410732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.410728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-dshm\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.410973 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.410773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf2eca8-f317-4e47-9909-e882d5882041-tls-certs\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.410973 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.410941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-tmp-dir\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.411060 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.411001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-home\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.411130 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.411104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-model-cache\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.412877 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.412848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-dshm\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.413136 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.413116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf2eca8-f317-4e47-9909-e882d5882041-tls-certs\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.418300 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.418276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpqb9\" (UniqueName: \"kubernetes.io/projected/caf2eca8-f317-4e47-9909-e882d5882041-kube-api-access-tpqb9\") pod \"distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") " pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.587316 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.587211    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:13.865986 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:13.865957    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"]
Sep 11 22:13:13.867497 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:13:13.867466    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaf2eca8_f317_4e47_9909_e882d5882041.slice/crio-0057520ff107d78f0fa1426d3943e2ef52d0e9f96332fe957c78a5f73c79584f WatchSource:0}: Error finding container 0057520ff107d78f0fa1426d3943e2ef52d0e9f96332fe957c78a5f73c79584f: Status 404 returned error can't find the container with id 0057520ff107d78f0fa1426d3943e2ef52d0e9f96332fe957c78a5f73c79584f
Sep 11 22:13:14.701101 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:14.701072    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:14.701475 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:14.701123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:14.713462 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:14.713436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:14.866826 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:14.866782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg" event={"ID":"caf2eca8-f317-4e47-9909-e882d5882041","Type":"ContainerStarted","Data":"deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c"}
Sep 11 22:13:14.866826 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:14.866828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg" event={"ID":"caf2eca8-f317-4e47-9909-e882d5882041","Type":"ContainerStarted","Data":"0057520ff107d78f0fa1426d3943e2ef52d0e9f96332fe957c78a5f73c79584f"}
Sep 11 22:13:14.878073 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:14.878047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:13:14.881653 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:14.881597    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg" podStartSLOduration=1.881582401 podStartE2EDuration="1.881582401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:13:14.880756742 +0000 UTC m=+1256.565703466" watchObservedRunningTime="2026-09-11 22:13:14.881582401 +0000 UTC m=+1256.566529148"
Sep 11 22:13:18.972263 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:18.972231    2578 scope.go:117] "RemoveContainer" containerID="b0aa4e0f72029f125f33960206fa956b7049202e60b11ecc99f23738f9469246"
Sep 11 22:13:23.588174 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:23.588131    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:23.588605 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:23.588282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:23.600810 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:23.600783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:13:23.937602 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:13:23.937574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:14:01.497339 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:01.497289    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"]
Sep 11 22:14:09.326813 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.326773    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"]
Sep 11 22:14:09.330621 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.330601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"
Sep 11 22:14:09.332716 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.332693    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:14:09.337001 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.336975    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"]
Sep 11 22:14:09.482162 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.482125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4383c27a-46e3-42f5-a49f-fa65c21deb56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv\" (UID: \"4383c27a-46e3-42f5-a49f-fa65c21deb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"
Sep 11 22:14:09.583296 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.583215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4383c27a-46e3-42f5-a49f-fa65c21deb56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv\" (UID: \"4383c27a-46e3-42f5-a49f-fa65c21deb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"
Sep 11 22:14:09.583869 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.583850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4383c27a-46e3-42f5-a49f-fa65c21deb56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv\" (UID: \"4383c27a-46e3-42f5-a49f-fa65c21deb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"
Sep 11 22:14:09.642290 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.642257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"
Sep 11 22:14:09.918424 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:09.918395    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"]
Sep 11 22:14:09.920442 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:14:09.920414    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4383c27a_46e3_42f5_a49f_fa65c21deb56.slice/crio-f84666e584f56eec9debb23a725a34e6eef769ccdae434b9a7fcb47c9bc27be0 WatchSource:0}: Error finding container f84666e584f56eec9debb23a725a34e6eef769ccdae434b9a7fcb47c9bc27be0: Status 404 returned error can't find the container with id f84666e584f56eec9debb23a725a34e6eef769ccdae434b9a7fcb47c9bc27be0
Sep 11 22:14:10.210488 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:10.210448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv" event={"ID":"4383c27a-46e3-42f5-a49f-fa65c21deb56","Type":"ContainerStarted","Data":"f84666e584f56eec9debb23a725a34e6eef769ccdae434b9a7fcb47c9bc27be0"}
Sep 11 22:14:11.218842 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:11.218803    2578 generic.go:356] "Generic (PLEG): container finished" podID="4383c27a-46e3-42f5-a49f-fa65c21deb56" containerID="cf7448b8107d6ef74348eb0dd02cc1c76aa23d96b88f14172297f437e3e6d9f4" exitCode=0
Sep 11 22:14:11.219278 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:11.218890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv" event={"ID":"4383c27a-46e3-42f5-a49f-fa65c21deb56","Type":"ContainerDied","Data":"cf7448b8107d6ef74348eb0dd02cc1c76aa23d96b88f14172297f437e3e6d9f4"}
Sep 11 22:14:11.588554 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:11.588471    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"]
Sep 11 22:14:12.367137 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:12.367110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"
Sep 11 22:14:12.514503 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:12.514407    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4383c27a-46e3-42f5-a49f-fa65c21deb56-maas-api-service-ca\") pod \"4383c27a-46e3-42f5-a49f-fa65c21deb56\" (UID: \"4383c27a-46e3-42f5-a49f-fa65c21deb56\") "
Sep 11 22:14:12.514803 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:12.514777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4383c27a-46e3-42f5-a49f-fa65c21deb56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4383c27a-46e3-42f5-a49f-fa65c21deb56" (UID: "4383c27a-46e3-42f5-a49f-fa65c21deb56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:14:12.514891 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:12.514871    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4383c27a-46e3-42f5-a49f-fa65c21deb56-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:13.232830 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:13.232797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv" event={"ID":"4383c27a-46e3-42f5-a49f-fa65c21deb56","Type":"ContainerDied","Data":"f84666e584f56eec9debb23a725a34e6eef769ccdae434b9a7fcb47c9bc27be0"}
Sep 11 22:14:13.232830 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:13.232821    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"
Sep 11 22:14:13.233042 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:13.232844    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f84666e584f56eec9debb23a725a34e6eef769ccdae434b9a7fcb47c9bc27be0"
Sep 11 22:14:16.516952 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.516879    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x" podUID="63a36b2e-3feb-45bb-bbb0-45eed2987393" containerName="main" containerID="cri-o://6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f" gracePeriod=45
Sep 11 22:14:16.778104 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.778042    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:14:16.963332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a36b2e-3feb-45bb-bbb0-45eed2987393-tls-certs\") pod \"63a36b2e-3feb-45bb-bbb0-45eed2987393\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") "
Sep 11 22:14:16.963534 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963341    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-home\") pod \"63a36b2e-3feb-45bb-bbb0-45eed2987393\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") "
Sep 11 22:14:16.963534 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-model-cache\") pod \"63a36b2e-3feb-45bb-bbb0-45eed2987393\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") "
Sep 11 22:14:16.963534 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttccz\" (UniqueName: \"kubernetes.io/projected/63a36b2e-3feb-45bb-bbb0-45eed2987393-kube-api-access-ttccz\") pod \"63a36b2e-3feb-45bb-bbb0-45eed2987393\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") "
Sep 11 22:14:16.963534 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-tmp-dir\") pod \"63a36b2e-3feb-45bb-bbb0-45eed2987393\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") "
Sep 11 22:14:16.963534 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-dshm\") pod \"63a36b2e-3feb-45bb-bbb0-45eed2987393\" (UID: \"63a36b2e-3feb-45bb-bbb0-45eed2987393\") "
Sep 11 22:14:16.963830 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-home" (OuterVolumeSpecName: "home") pod "63a36b2e-3feb-45bb-bbb0-45eed2987393" (UID: "63a36b2e-3feb-45bb-bbb0-45eed2987393"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:16.963830 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963725    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-model-cache" (OuterVolumeSpecName: "model-cache") pod "63a36b2e-3feb-45bb-bbb0-45eed2987393" (UID: "63a36b2e-3feb-45bb-bbb0-45eed2987393"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:16.963830 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963790    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63a36b2e-3feb-45bb-bbb0-45eed2987393" (UID: "63a36b2e-3feb-45bb-bbb0-45eed2987393"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:16.963944 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963935    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:16.963983 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963951    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:16.963983 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.963979    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:16.965720 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.965695    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-dshm" (OuterVolumeSpecName: "dshm") pod "63a36b2e-3feb-45bb-bbb0-45eed2987393" (UID: "63a36b2e-3feb-45bb-bbb0-45eed2987393"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:16.966026 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.965999    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a36b2e-3feb-45bb-bbb0-45eed2987393-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63a36b2e-3feb-45bb-bbb0-45eed2987393" (UID: "63a36b2e-3feb-45bb-bbb0-45eed2987393"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:14:16.966026 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:16.966008    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a36b2e-3feb-45bb-bbb0-45eed2987393-kube-api-access-ttccz" (OuterVolumeSpecName: "kube-api-access-ttccz") pod "63a36b2e-3feb-45bb-bbb0-45eed2987393" (UID: "63a36b2e-3feb-45bb-bbb0-45eed2987393"). InnerVolumeSpecName "kube-api-access-ttccz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:14:17.065499 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.065395    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttccz\" (UniqueName: \"kubernetes.io/projected/63a36b2e-3feb-45bb-bbb0-45eed2987393-kube-api-access-ttccz\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:17.065499 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.065431    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a36b2e-3feb-45bb-bbb0-45eed2987393-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:17.065499 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.065445    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a36b2e-3feb-45bb-bbb0-45eed2987393-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:17.259015 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.258974    2578 generic.go:356] "Generic (PLEG): container finished" podID="63a36b2e-3feb-45bb-bbb0-45eed2987393" containerID="6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f" exitCode=0
Sep 11 22:14:17.259182 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.259037    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"
Sep 11 22:14:17.259182 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.259060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x" event={"ID":"63a36b2e-3feb-45bb-bbb0-45eed2987393","Type":"ContainerDied","Data":"6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f"}
Sep 11 22:14:17.259182 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.259115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x" event={"ID":"63a36b2e-3feb-45bb-bbb0-45eed2987393","Type":"ContainerDied","Data":"3136f96c832f43235fc0d0542623e640f880b164ca3701d78da356e6484ac2a6"}
Sep 11 22:14:17.259182 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.259138    2578 scope.go:117] "RemoveContainer" containerID="6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f"
Sep 11 22:14:17.269707 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.269688    2578 scope.go:117] "RemoveContainer" containerID="6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f"
Sep 11 22:14:17.270031 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:14:17.270010    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f\": container with ID starting with 6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f not found: ID does not exist" containerID="6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f"
Sep 11 22:14:17.270102 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.270039    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f"} err="failed to get container status \"6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f\": rpc error: code = NotFound desc = could not find container \"6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f\": container with ID starting with 6d8ca63b0444b1eaa65a4c57dd0b1f7c749739e81bcba6389624f6ebe07b2a9f not found: ID does not exist"
Sep 11 22:14:17.279910 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.279885    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"]
Sep 11 22:14:17.286585 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:17.286564    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-b-f25d30a7-kserve-66c45d84cc-gnl7x"]
Sep 11 22:14:18.857891 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:18.857855    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63a36b2e-3feb-45bb-bbb0-45eed2987393" path="/var/lib/kubelet/pods/63a36b2e-3feb-45bb-bbb0-45eed2987393/volumes"
Sep 11 22:14:22.171094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.171048    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"]
Sep 11 22:14:22.171703 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.171684    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63a36b2e-3feb-45bb-bbb0-45eed2987393" containerName="main"
Sep 11 22:14:22.171814 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.171706    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="63a36b2e-3feb-45bb-bbb0-45eed2987393" containerName="main"
Sep 11 22:14:22.171814 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.171727    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4383c27a-46e3-42f5-a49f-fa65c21deb56" containerName="revoke-keys"
Sep 11 22:14:22.171814 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.171735    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4383c27a-46e3-42f5-a49f-fa65c21deb56" containerName="revoke-keys"
Sep 11 22:14:22.171965 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.171862    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4383c27a-46e3-42f5-a49f-fa65c21deb56" containerName="revoke-keys"
Sep 11 22:14:22.171965 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.171881    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="63a36b2e-3feb-45bb-bbb0-45eed2987393" containerName="main"
Sep 11 22:14:22.176177 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.176155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.178436 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.178412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"distinct-w4-7e2add-kserve-self-signed-certs\""
Sep 11 22:14:22.185087 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.185054    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"]
Sep 11 22:14:22.215267 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.215230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tls-certs\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.215267 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.215267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-home\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.215465 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.215296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-268kf\" (UniqueName: \"kubernetes.io/projected/86b9512f-6a84-439e-8ebd-51d8d18f5e88-kube-api-access-268kf\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.215465 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.215407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tmp-dir\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.215465 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.215459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-model-cache\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.215565 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.215502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-dshm\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.316962 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.316927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tmp-dir\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.317120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.316981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-model-cache\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.317120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.317012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-dshm\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.317120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.317055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tls-certs\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.317120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.317072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-home\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.317120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.317102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-268kf\" (UniqueName: \"kubernetes.io/projected/86b9512f-6a84-439e-8ebd-51d8d18f5e88-kube-api-access-268kf\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.317420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.317330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tmp-dir\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.317420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.317383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-model-cache\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.317528 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.317458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-home\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.319393 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.319365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-dshm\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.319532 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.319402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tls-certs\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.324698 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.324679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-268kf\" (UniqueName: \"kubernetes.io/projected/86b9512f-6a84-439e-8ebd-51d8d18f5e88-kube-api-access-268kf\") pod \"distinct-w4-7e2add-kserve-c8dd594bb-kg85m\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.487129 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.487037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:22.759824 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:22.759738    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"]
Sep 11 22:14:22.762798 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:14:22.762772    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86b9512f_6a84_439e_8ebd_51d8d18f5e88.slice/crio-b67f7e21faec5bfbadb7dac3b852043c579b218751b61bfb95a39b9a037686a5 WatchSource:0}: Error finding container b67f7e21faec5bfbadb7dac3b852043c579b218751b61bfb95a39b9a037686a5: Status 404 returned error can't find the container with id b67f7e21faec5bfbadb7dac3b852043c579b218751b61bfb95a39b9a037686a5
Sep 11 22:14:23.307833 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:23.307797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m" event={"ID":"86b9512f-6a84-439e-8ebd-51d8d18f5e88","Type":"ContainerStarted","Data":"a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d"}
Sep 11 22:14:23.307833 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:23.307841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m" event={"ID":"86b9512f-6a84-439e-8ebd-51d8d18f5e88","Type":"ContainerStarted","Data":"b67f7e21faec5bfbadb7dac3b852043c579b218751b61bfb95a39b9a037686a5"}
Sep 11 22:14:23.322246 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:23.322162    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m" podStartSLOduration=1.322143347 podStartE2EDuration="1.322143347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:14:23.321921107 +0000 UTC m=+1325.006867834" watchObservedRunningTime="2026-09-11 22:14:23.322143347 +0000 UTC m=+1325.007090062"
Sep 11 22:14:24.921656 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:24.919970    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"]
Sep 11 22:14:24.921656 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:24.920322    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c" podUID="ecd4d2fa-48bd-40a2-bd76-7246bb7677d8" containerName="payload-pre-processing" containerID="cri-o://ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61" gracePeriod=30
Sep 11 22:14:25.088352 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.088303    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"]
Sep 11 22:14:25.090830 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.090804    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92d3298e-gqqxv"]
Sep 11 22:14:25.207406 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.207375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:14:25.246945 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.246914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-plugins-config-volume\") pod \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\" (UID: \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\") "
Sep 11 22:14:25.247112 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.247007    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htdnn\" (UniqueName: \"kubernetes.io/projected/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-kube-api-access-htdnn\") pod \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\" (UID: \"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8\") "
Sep 11 22:14:25.247379 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.247263    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecd4d2fa-48bd-40a2-bd76-7246bb7677d8" (UID: "ecd4d2fa-48bd-40a2-bd76-7246bb7677d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:14:25.249189 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.249170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-kube-api-access-htdnn" (OuterVolumeSpecName: "kube-api-access-htdnn") pod "ecd4d2fa-48bd-40a2-bd76-7246bb7677d8" (UID: "ecd4d2fa-48bd-40a2-bd76-7246bb7677d8"). InnerVolumeSpecName "kube-api-access-htdnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:14:25.322588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.322547    2578 generic.go:356] "Generic (PLEG): container finished" podID="ecd4d2fa-48bd-40a2-bd76-7246bb7677d8" containerID="ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61" exitCode=0
Sep 11 22:14:25.322771 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.322595    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"
Sep 11 22:14:25.322771 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.322647    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c" event={"ID":"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8","Type":"ContainerDied","Data":"ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61"}
Sep 11 22:14:25.322771 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.322706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c" event={"ID":"ecd4d2fa-48bd-40a2-bd76-7246bb7677d8","Type":"ContainerDied","Data":"7b9cbfd768b571fa0dfb32b7b54b175065fbf80ada070a248f20df86235ead43"}
Sep 11 22:14:25.322771 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.322735    2578 scope.go:117] "RemoveContainer" containerID="ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61"
Sep 11 22:14:25.332122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.332100    2578 scope.go:117] "RemoveContainer" containerID="ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61"
Sep 11 22:14:25.332387 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:14:25.332370    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61\": container with ID starting with ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61 not found: ID does not exist" containerID="ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61"
Sep 11 22:14:25.332432 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.332395    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61"} err="failed to get container status \"ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61\": rpc error: code = NotFound desc = could not find container \"ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61\": container with ID starting with ad8103064b56e3e3b19f2104320b071b24613c89a15f9fd3800aa623d81b4b61 not found: ID does not exist"
Sep 11 22:14:25.343343 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.343314    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"]
Sep 11 22:14:25.346466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.346440    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-92d3298e-67d5764c4b-vp95c"]
Sep 11 22:14:25.348434 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.348414    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:25.348434 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:25.348435    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htdnn\" (UniqueName: \"kubernetes.io/projected/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8-kube-api-access-htdnn\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:26.611530 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.611481    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n" podUID="17c6cded-5048-4ca9-9a74-ba1457144420" containerName="main" containerID="cri-o://2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb" gracePeriod=45
Sep 11 22:14:26.859425 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.859394    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4383c27a-46e3-42f5-a49f-fa65c21deb56" path="/var/lib/kubelet/pods/4383c27a-46e3-42f5-a49f-fa65c21deb56/volumes"
Sep 11 22:14:26.859895 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.859869    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd4d2fa-48bd-40a2-bd76-7246bb7677d8" path="/var/lib/kubelet/pods/ecd4d2fa-48bd-40a2-bd76-7246bb7677d8/volumes"
Sep 11 22:14:26.864253 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.864206    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:14:26.964590 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.964552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-dshm\") pod \"17c6cded-5048-4ca9-9a74-ba1457144420\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") "
Sep 11 22:14:26.964784 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.964613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggr76\" (UniqueName: \"kubernetes.io/projected/17c6cded-5048-4ca9-9a74-ba1457144420-kube-api-access-ggr76\") pod \"17c6cded-5048-4ca9-9a74-ba1457144420\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") "
Sep 11 22:14:26.964784 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.964724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-model-cache\") pod \"17c6cded-5048-4ca9-9a74-ba1457144420\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") "
Sep 11 22:14:26.964784 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.964756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-tmp-dir\") pod \"17c6cded-5048-4ca9-9a74-ba1457144420\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") "
Sep 11 22:14:26.964961 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.964806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c6cded-5048-4ca9-9a74-ba1457144420-tls-certs\") pod \"17c6cded-5048-4ca9-9a74-ba1457144420\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") "
Sep 11 22:14:26.964961 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.964872    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-home\") pod \"17c6cded-5048-4ca9-9a74-ba1457144420\" (UID: \"17c6cded-5048-4ca9-9a74-ba1457144420\") "
Sep 11 22:14:26.965065 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.965030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17c6cded-5048-4ca9-9a74-ba1457144420" (UID: "17c6cded-5048-4ca9-9a74-ba1457144420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:26.965267 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.965016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-model-cache" (OuterVolumeSpecName: "model-cache") pod "17c6cded-5048-4ca9-9a74-ba1457144420" (UID: "17c6cded-5048-4ca9-9a74-ba1457144420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:26.965400 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.965239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-home" (OuterVolumeSpecName: "home") pod "17c6cded-5048-4ca9-9a74-ba1457144420" (UID: "17c6cded-5048-4ca9-9a74-ba1457144420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:26.965400 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.965381    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:26.965400 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.965394    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:26.966857 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.966838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c6cded-5048-4ca9-9a74-ba1457144420-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17c6cded-5048-4ca9-9a74-ba1457144420" (UID: "17c6cded-5048-4ca9-9a74-ba1457144420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:14:26.967036 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.967022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-dshm" (OuterVolumeSpecName: "dshm") pod "17c6cded-5048-4ca9-9a74-ba1457144420" (UID: "17c6cded-5048-4ca9-9a74-ba1457144420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:26.967100 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:26.967053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c6cded-5048-4ca9-9a74-ba1457144420-kube-api-access-ggr76" (OuterVolumeSpecName: "kube-api-access-ggr76") pod "17c6cded-5048-4ca9-9a74-ba1457144420" (UID: "17c6cded-5048-4ca9-9a74-ba1457144420"). InnerVolumeSpecName "kube-api-access-ggr76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:14:27.066150 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.066114    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c6cded-5048-4ca9-9a74-ba1457144420-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:27.066150 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.066146    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:27.066150 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.066154    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c6cded-5048-4ca9-9a74-ba1457144420-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:27.066394 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.066163    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggr76\" (UniqueName: \"kubernetes.io/projected/17c6cded-5048-4ca9-9a74-ba1457144420-kube-api-access-ggr76\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:27.336892 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.336858    2578 generic.go:356] "Generic (PLEG): container finished" podID="17c6cded-5048-4ca9-9a74-ba1457144420" containerID="2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb" exitCode=0
Sep 11 22:14:27.337081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.336900    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"
Sep 11 22:14:27.337081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.336947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n" event={"ID":"17c6cded-5048-4ca9-9a74-ba1457144420","Type":"ContainerDied","Data":"2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb"}
Sep 11 22:14:27.337081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.336991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n" event={"ID":"17c6cded-5048-4ca9-9a74-ba1457144420","Type":"ContainerDied","Data":"8c353ae7f4781dbac6735ebbd1d7a957e394e0ce4d5d53cb4e20281d537e2536"}
Sep 11 22:14:27.337081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.337010    2578 scope.go:117] "RemoveContainer" containerID="2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb"
Sep 11 22:14:27.346995 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.346977    2578 scope.go:117] "RemoveContainer" containerID="2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb"
Sep 11 22:14:27.347278 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:14:27.347260    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb\": container with ID starting with 2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb not found: ID does not exist" containerID="2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb"
Sep 11 22:14:27.347328 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.347287    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb"} err="failed to get container status \"2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb\": rpc error: code = NotFound desc = could not find container \"2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb\": container with ID starting with 2236bf37aad9a541c6b2a1f324e2b60b46db98ddb1c24e906c18744e49cc89eb not found: ID does not exist"
Sep 11 22:14:27.358326 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.358299    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"]
Sep 11 22:14:27.362150 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:27.362122    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/e2e-conflict-a-f25d30a7-kserve-7f56f5dbff-wpg8n"]
Sep 11 22:14:28.858941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:28.858907    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c6cded-5048-4ca9-9a74-ba1457144420" path="/var/lib/kubelet/pods/17c6cded-5048-4ca9-9a74-ba1457144420/volumes"
Sep 11 22:14:32.488203 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.488150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:32.488203 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.488215    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:32.500809 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.500783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:32.987691 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.987658    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"]
Sep 11 22:14:32.988179 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.988155    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd4d2fa-48bd-40a2-bd76-7246bb7677d8" containerName="payload-pre-processing"
Sep 11 22:14:32.988179 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.988177    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd4d2fa-48bd-40a2-bd76-7246bb7677d8" containerName="payload-pre-processing"
Sep 11 22:14:32.988357 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.988186    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c6cded-5048-4ca9-9a74-ba1457144420" containerName="main"
Sep 11 22:14:32.988357 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.988192    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c6cded-5048-4ca9-9a74-ba1457144420" containerName="main"
Sep 11 22:14:32.988357 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.988262    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd4d2fa-48bd-40a2-bd76-7246bb7677d8" containerName="payload-pre-processing"
Sep 11 22:14:32.988357 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.988269    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c6cded-5048-4ca9-9a74-ba1457144420" containerName="main"
Sep 11 22:14:32.992574 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.992558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:32.995335 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.995307    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"kube-root-ca.crt\""
Sep 11 22:14:32.995335 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.995324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"auth-test-model-966527-kserve-self-signed-certs\""
Sep 11 22:14:32.995506 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.995382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"default-dockercfg-tc5dd\""
Sep 11 22:14:32.995506 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:32.995394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"openshift-service-ca.crt\""
Sep 11 22:14:33.002456 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.001945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"]
Sep 11 22:14:33.029452 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.029417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-home\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.029599 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.029465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-model-cache\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.029599 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.029492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-dshm\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.029599 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.029522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lt75\" (UniqueName: \"kubernetes.io/projected/b967773c-936d-4e3a-82cc-5c9a7040dc67-kube-api-access-7lt75\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.029599 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.029552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-tmp-dir\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.029786 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.029629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b967773c-936d-4e3a-82cc-5c9a7040dc67-tls-certs\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.130813 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.130776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b967773c-936d-4e3a-82cc-5c9a7040dc67-tls-certs\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.131017 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.130831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-home\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.131017 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.130858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-model-cache\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.131017 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.130876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-dshm\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.131017 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.130893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lt75\" (UniqueName: \"kubernetes.io/projected/b967773c-936d-4e3a-82cc-5c9a7040dc67-kube-api-access-7lt75\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.131017 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.130921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-tmp-dir\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.131386 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.131360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-model-cache\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.131465 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.131375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-home\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.131465 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.131430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-tmp-dir\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.133220 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.133197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-dshm\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.133350 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.133333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b967773c-936d-4e3a-82cc-5c9a7040dc67-tls-certs\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.138055 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.138027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lt75\" (UniqueName: \"kubernetes.io/projected/b967773c-936d-4e3a-82cc-5c9a7040dc67-kube-api-access-7lt75\") pod \"auth-test-model-966527-kserve-78764fdf8c-vvwhj\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") " pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.192545 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.192512    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"]
Sep 11 22:14:33.305725 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.305595    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:33.389525 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.389491    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:14:33.581668 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:33.581624    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"]
Sep 11 22:14:33.583045 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:14:33.583020    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb967773c_936d_4e3a_82cc_5c9a7040dc67.slice/crio-5438df968dc0bcf3ee5e25e79da9d27b972d91033acb69e230e2d4fb8aebaf9c WatchSource:0}: Error finding container 5438df968dc0bcf3ee5e25e79da9d27b972d91033acb69e230e2d4fb8aebaf9c: Status 404 returned error can't find the container with id 5438df968dc0bcf3ee5e25e79da9d27b972d91033acb69e230e2d4fb8aebaf9c
Sep 11 22:14:34.386004 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:34.385967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj" event={"ID":"b967773c-936d-4e3a-82cc-5c9a7040dc67","Type":"ContainerStarted","Data":"2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f"}
Sep 11 22:14:34.386192 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:34.386019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj" event={"ID":"b967773c-936d-4e3a-82cc-5c9a7040dc67","Type":"ContainerStarted","Data":"5438df968dc0bcf3ee5e25e79da9d27b972d91033acb69e230e2d4fb8aebaf9c"}
Sep 11 22:14:34.405199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:34.405139    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj" podStartSLOduration=2.405121349 podStartE2EDuration="2.405121349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:14:34.402810566 +0000 UTC m=+1336.087757283" watchObservedRunningTime="2026-09-11 22:14:34.405121349 +0000 UTC m=+1336.090068063"
Sep 11 22:14:43.306048 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:43.305945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:43.306048 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:43.306034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:43.318468 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:43.318443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:43.448710 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:43.448684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:14:48.211842 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.211793    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz" podUID="e4d07f2e-b344-4101-b706-9b35f8f238bc" containerName="main" containerID="cri-o://a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995" gracePeriod=45
Sep 11 22:14:48.441461 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.441435    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:14:48.469360 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.469275    2578 generic.go:356] "Generic (PLEG): container finished" podID="e4d07f2e-b344-4101-b706-9b35f8f238bc" containerID="a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995" exitCode=0
Sep 11 22:14:48.469360 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.469322    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"
Sep 11 22:14:48.469571 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.469358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz" event={"ID":"e4d07f2e-b344-4101-b706-9b35f8f238bc","Type":"ContainerDied","Data":"a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995"}
Sep 11 22:14:48.469571 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.469410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz" event={"ID":"e4d07f2e-b344-4101-b706-9b35f8f238bc","Type":"ContainerDied","Data":"2b33247f32f288e92d0d3e872693730515b50e50e0d26f5b7de90629c165f349"}
Sep 11 22:14:48.469571 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.469431    2578 scope.go:117] "RemoveContainer" containerID="a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995"
Sep 11 22:14:48.483185 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.483163    2578 scope.go:117] "RemoveContainer" containerID="a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995"
Sep 11 22:14:48.483517 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:14:48.483489    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995\": container with ID starting with a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995 not found: ID does not exist" containerID="a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995"
Sep 11 22:14:48.483586 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.483530    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995"} err="failed to get container status \"a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995\": rpc error: code = NotFound desc = could not find container \"a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995\": container with ID starting with a2d329310014a97e1b8daa5014d0db0715f0061568fdbea807d5a02bee340995 not found: ID does not exist"
Sep 11 22:14:48.599978 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.599944    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-model-cache\") pod \"e4d07f2e-b344-4101-b706-9b35f8f238bc\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") "
Sep 11 22:14:48.600167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-tmp-dir\") pod \"e4d07f2e-b344-4101-b706-9b35f8f238bc\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") "
Sep 11 22:14:48.600167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600080    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-home\") pod \"e4d07f2e-b344-4101-b706-9b35f8f238bc\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") "
Sep 11 22:14:48.600167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6prj\" (UniqueName: \"kubernetes.io/projected/e4d07f2e-b344-4101-b706-9b35f8f238bc-kube-api-access-l6prj\") pod \"e4d07f2e-b344-4101-b706-9b35f8f238bc\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") "
Sep 11 22:14:48.600167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-dshm\") pod \"e4d07f2e-b344-4101-b706-9b35f8f238bc\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") "
Sep 11 22:14:48.600167 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600140    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d07f2e-b344-4101-b706-9b35f8f238bc-tls-certs\") pod \"e4d07f2e-b344-4101-b706-9b35f8f238bc\" (UID: \"e4d07f2e-b344-4101-b706-9b35f8f238bc\") "
Sep 11 22:14:48.600443 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600244    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4d07f2e-b344-4101-b706-9b35f8f238bc" (UID: "e4d07f2e-b344-4101-b706-9b35f8f238bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:48.600443 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4d07f2e-b344-4101-b706-9b35f8f238bc" (UID: "e4d07f2e-b344-4101-b706-9b35f8f238bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:48.600443 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600346    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-home" (OuterVolumeSpecName: "home") pod "e4d07f2e-b344-4101-b706-9b35f8f238bc" (UID: "e4d07f2e-b344-4101-b706-9b35f8f238bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:48.600593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600474    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:48.600593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600492    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:48.600593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.600507    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:48.602402 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.602378    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-dshm" (OuterVolumeSpecName: "dshm") pod "e4d07f2e-b344-4101-b706-9b35f8f238bc" (UID: "e4d07f2e-b344-4101-b706-9b35f8f238bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:14:48.602521 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.602453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d07f2e-b344-4101-b706-9b35f8f238bc-kube-api-access-l6prj" (OuterVolumeSpecName: "kube-api-access-l6prj") pod "e4d07f2e-b344-4101-b706-9b35f8f238bc" (UID: "e4d07f2e-b344-4101-b706-9b35f8f238bc"). InnerVolumeSpecName "kube-api-access-l6prj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:14:48.602521 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.602505    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d07f2e-b344-4101-b706-9b35f8f238bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4d07f2e-b344-4101-b706-9b35f8f238bc" (UID: "e4d07f2e-b344-4101-b706-9b35f8f238bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:14:48.701218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.701178    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6prj\" (UniqueName: \"kubernetes.io/projected/e4d07f2e-b344-4101-b706-9b35f8f238bc-kube-api-access-l6prj\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:48.701218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.701209    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d07f2e-b344-4101-b706-9b35f8f238bc-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:48.701218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.701222    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d07f2e-b344-4101-b706-9b35f8f238bc-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:14:48.804152 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.804118    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"]
Sep 11 22:14:48.819425 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.819390    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-92d3298e/e2e-lifecycle-model-92d3298e-kserve-695b6c8c5-jfbtz"]
Sep 11 22:14:48.857703 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:14:48.857623    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d07f2e-b344-4101-b706-9b35f8f238bc" path="/var/lib/kubelet/pods/e4d07f2e-b344-4101-b706-9b35f8f238bc/volumes"
Sep 11 22:15:00.160821 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.160779    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"]
Sep 11 22:15:00.161419 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.161396    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d07f2e-b344-4101-b706-9b35f8f238bc" containerName="main"
Sep 11 22:15:00.161504 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.161422    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d07f2e-b344-4101-b706-9b35f8f238bc" containerName="main"
Sep 11 22:15:00.161560 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.161550    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d07f2e-b344-4101-b706-9b35f8f238bc" containerName="main"
Sep 11 22:15:00.165870 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.165847    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"]
Sep 11 22:15:00.165989 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.165967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"
Sep 11 22:15:00.167895 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.167870    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:15:00.169451 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.169427    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"]
Sep 11 22:15:00.169570 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.169540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"
Sep 11 22:15:00.172876 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.172862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"
Sep 11 22:15:00.178178 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.178161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"
Sep 11 22:15:00.189616 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.189580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"]
Sep 11 22:15:00.192183 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.192158    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"
Sep 11 22:15:00.193820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.193794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"]
Sep 11 22:15:00.198606 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.198133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"]
Sep 11 22:15:00.201926 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.201906    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"]
Sep 11 22:15:00.206950 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.206931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"
Sep 11 22:15:00.214221 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.214192    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"]
Sep 11 22:15:00.246964 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.246935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"
Sep 11 22:15:00.257887 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.256691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"
Sep 11 22:15:00.472577 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.472550    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"]
Sep 11 22:15:00.475904 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:15:00.475864    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c50d293_982b_42a1_88ac_df758a1504f8.slice/crio-6353de4a399f29cd72e168ad81e709e0c8cfcce8a73b410bba9db6020dab6ca6 WatchSource:0}: Error finding container 6353de4a399f29cd72e168ad81e709e0c8cfcce8a73b410bba9db6020dab6ca6: Status 404 returned error can't find the container with id 6353de4a399f29cd72e168ad81e709e0c8cfcce8a73b410bba9db6020dab6ca6
Sep 11 22:15:00.495040 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.495016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"]
Sep 11 22:15:00.496046 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:15:00.496022    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58959601_b65d_4f5d_b067_96c6a5e293db.slice/crio-4cda696f5162535d3dae5dc032213c6acabb56622548b0c54308332791706c01 WatchSource:0}: Error finding container 4cda696f5162535d3dae5dc032213c6acabb56622548b0c54308332791706c01: Status 404 returned error can't find the container with id 4cda696f5162535d3dae5dc032213c6acabb56622548b0c54308332791706c01
Sep 11 22:15:00.545533 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.545498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc" event={"ID":"58959601-b65d-4f5d-b067-96c6a5e293db","Type":"ContainerStarted","Data":"4cda696f5162535d3dae5dc032213c6acabb56622548b0c54308332791706c01"}
Sep 11 22:15:00.546816 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.546792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng" event={"ID":"9c50d293-982b-42a1-88ac-df758a1504f8","Type":"ContainerStarted","Data":"6353de4a399f29cd72e168ad81e709e0c8cfcce8a73b410bba9db6020dab6ca6"}
Sep 11 22:15:00.558730 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.558660    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"]
Sep 11 22:15:00.561607 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:15:00.561584    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc893eb_06ce_4ac8_af38_adf8817ace84.slice/crio-e432a377ff7b315e715134aa4859c90a7bc2e1f7600ed09148fa93530af39a93 WatchSource:0}: Error finding container e432a377ff7b315e715134aa4859c90a7bc2e1f7600ed09148fa93530af39a93: Status 404 returned error can't find the container with id e432a377ff7b315e715134aa4859c90a7bc2e1f7600ed09148fa93530af39a93
Sep 11 22:15:00.562951 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:00.562932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"]
Sep 11 22:15:00.564513 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:15:00.564492    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48bf43b_468d_4895_9e33_0c624c6df8a5.slice/crio-e4178a4b76758623e3219bff77d591700ee6e8eb16005e12aba034601d481a0d WatchSource:0}: Error finding container e4178a4b76758623e3219bff77d591700ee6e8eb16005e12aba034601d481a0d: Status 404 returned error can't find the container with id e4178a4b76758623e3219bff77d591700ee6e8eb16005e12aba034601d481a0d
Sep 11 22:15:01.554248 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.554205    2578 generic.go:356] "Generic (PLEG): container finished" podID="2bc893eb-06ce-4ac8-af38-adf8817ace84" containerID="56f4bfa2b2b43b0f493908dad4935c34028ff7c4dfeb7eb6aa2753ea2c508c55" exitCode=0
Sep 11 22:15:01.554727 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.554287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4" event={"ID":"2bc893eb-06ce-4ac8-af38-adf8817ace84","Type":"ContainerDied","Data":"56f4bfa2b2b43b0f493908dad4935c34028ff7c4dfeb7eb6aa2753ea2c508c55"}
Sep 11 22:15:01.554727 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.554344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4" event={"ID":"2bc893eb-06ce-4ac8-af38-adf8817ace84","Type":"ContainerStarted","Data":"e432a377ff7b315e715134aa4859c90a7bc2e1f7600ed09148fa93530af39a93"}
Sep 11 22:15:01.555980 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.555959    2578 generic.go:356] "Generic (PLEG): container finished" podID="58959601-b65d-4f5d-b067-96c6a5e293db" containerID="f6dce56c46719160ed8b9dd2527f1854a8459cdc6eeaa07ee23c22bb29f610b8" exitCode=0
Sep 11 22:15:01.556107 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.556036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc" event={"ID":"58959601-b65d-4f5d-b067-96c6a5e293db","Type":"ContainerDied","Data":"f6dce56c46719160ed8b9dd2527f1854a8459cdc6eeaa07ee23c22bb29f610b8"}
Sep 11 22:15:01.557521 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.557499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d" event={"ID":"e48bf43b-468d-4895-9e33-0c624c6df8a5","Type":"ContainerStarted","Data":"d9f0d48a27ad6275f2102c9d9a0b63c5a56b35649c946982a4e675504ef88806"}
Sep 11 22:15:01.557622 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.557537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d" event={"ID":"e48bf43b-468d-4895-9e33-0c624c6df8a5","Type":"ContainerStarted","Data":"e4178a4b76758623e3219bff77d591700ee6e8eb16005e12aba034601d481a0d"}
Sep 11 22:15:01.559055 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.559030    2578 generic.go:356] "Generic (PLEG): container finished" podID="9c50d293-982b-42a1-88ac-df758a1504f8" containerID="bf7b03d09528795b0a24069c56ada3801898e6db7b2c3c55f3040f870d2cea66" exitCode=0
Sep 11 22:15:01.559147 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.559108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng" event={"ID":"9c50d293-982b-42a1-88ac-df758a1504f8","Type":"ContainerDied","Data":"bf7b03d09528795b0a24069c56ada3801898e6db7b2c3c55f3040f870d2cea66"}
Sep 11 22:15:01.685096 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:01.685042    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d" podStartSLOduration=1.685025883 podStartE2EDuration="1.685025883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:15:01.683751706 +0000 UTC m=+1363.368698420" watchObservedRunningTime="2026-09-11 22:15:01.685025883 +0000 UTC m=+1363.369972598"
Sep 11 22:15:02.567427 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:02.567388    2578 generic.go:356] "Generic (PLEG): container finished" podID="e48bf43b-468d-4895-9e33-0c624c6df8a5" containerID="d9f0d48a27ad6275f2102c9d9a0b63c5a56b35649c946982a4e675504ef88806" exitCode=0
Sep 11 22:15:02.567918 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:02.567472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d" event={"ID":"e48bf43b-468d-4895-9e33-0c624c6df8a5","Type":"ContainerDied","Data":"d9f0d48a27ad6275f2102c9d9a0b63c5a56b35649c946982a4e675504ef88806"}
Sep 11 22:15:02.762442 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:02.762415    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"
Sep 11 22:15:02.765478 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:02.765459    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"
Sep 11 22:15:02.768731 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:02.768713    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"
Sep 11 22:15:03.575121 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.575089    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"
Sep 11 22:15:03.575535 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.575102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4" event={"ID":"2bc893eb-06ce-4ac8-af38-adf8817ace84","Type":"ContainerDied","Data":"e432a377ff7b315e715134aa4859c90a7bc2e1f7600ed09148fa93530af39a93"}
Sep 11 22:15:03.575535 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.575197    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e432a377ff7b315e715134aa4859c90a7bc2e1f7600ed09148fa93530af39a93"
Sep 11 22:15:03.576569 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.576541    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"
Sep 11 22:15:03.576713 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.576567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc" event={"ID":"58959601-b65d-4f5d-b067-96c6a5e293db","Type":"ContainerDied","Data":"4cda696f5162535d3dae5dc032213c6acabb56622548b0c54308332791706c01"}
Sep 11 22:15:03.576713 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.576615    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cda696f5162535d3dae5dc032213c6acabb56622548b0c54308332791706c01"
Sep 11 22:15:03.578120 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.578094    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"
Sep 11 22:15:03.578217 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.578150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng" event={"ID":"9c50d293-982b-42a1-88ac-df758a1504f8","Type":"ContainerDied","Data":"6353de4a399f29cd72e168ad81e709e0c8cfcce8a73b410bba9db6020dab6ca6"}
Sep 11 22:15:03.578217 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.578190    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6353de4a399f29cd72e168ad81e709e0c8cfcce8a73b410bba9db6020dab6ca6"
Sep 11 22:15:03.696266 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:03.696240    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"
Sep 11 22:15:04.586538 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:04.586508    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"
Sep 11 22:15:04.586538 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:04.586529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d" event={"ID":"e48bf43b-468d-4895-9e33-0c624c6df8a5","Type":"ContainerDied","Data":"e4178a4b76758623e3219bff77d591700ee6e8eb16005e12aba034601d481a0d"}
Sep 11 22:15:04.587012 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:04.586581    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4178a4b76758623e3219bff77d591700ee6e8eb16005e12aba034601d481a0d"
Sep 11 22:15:22.076899 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.076860    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"]
Sep 11 22:15:22.077332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077255    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c50d293-982b-42a1-88ac-df758a1504f8" containerName="cleanup"
Sep 11 22:15:22.077332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077267    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c50d293-982b-42a1-88ac-df758a1504f8" containerName="cleanup"
Sep 11 22:15:22.077332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077284    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc893eb-06ce-4ac8-af38-adf8817ace84" containerName="cleanup"
Sep 11 22:15:22.077332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077293    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc893eb-06ce-4ac8-af38-adf8817ace84" containerName="cleanup"
Sep 11 22:15:22.077332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077307    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58959601-b65d-4f5d-b067-96c6a5e293db" containerName="cleanup"
Sep 11 22:15:22.077332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077313    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58959601-b65d-4f5d-b067-96c6a5e293db" containerName="cleanup"
Sep 11 22:15:22.077332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077325    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48bf43b-468d-4895-9e33-0c624c6df8a5" containerName="cleanup"
Sep 11 22:15:22.077332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077331    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48bf43b-468d-4895-9e33-0c624c6df8a5" containerName="cleanup"
Sep 11 22:15:22.077667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077392    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c50d293-982b-42a1-88ac-df758a1504f8" containerName="cleanup"
Sep 11 22:15:22.077667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077399    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc893eb-06ce-4ac8-af38-adf8817ace84" containerName="cleanup"
Sep 11 22:15:22.077667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077407    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58959601-b65d-4f5d-b067-96c6a5e293db" containerName="cleanup"
Sep 11 22:15:22.077667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.077414    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48bf43b-468d-4895-9e33-0c624c6df8a5" containerName="cleanup"
Sep 11 22:15:22.081786 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.081767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.083761 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.083742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7e2add\"/\"distinct-2-w4-7e2add-kserve-self-signed-certs\""
Sep 11 22:15:22.089392 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.089367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"]
Sep 11 22:15:22.238932 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.238888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-model-cache\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.238932 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.238938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-home\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.239164 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.238958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67753ed0-12fe-4da5-a887-f8c9df884fa2-tls-certs\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.239164 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.239069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-tmp-dir\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.239252 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.239178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-dshm\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.239252 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.239215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/67753ed0-12fe-4da5-a887-f8c9df884fa2-kube-api-access-f7qpm\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.339848 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.339755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-dshm\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.339848 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.339804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/67753ed0-12fe-4da5-a887-f8c9df884fa2-kube-api-access-f7qpm\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.339848 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.339830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-model-cache\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.339848 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.339850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-home\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.340192 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.339870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67753ed0-12fe-4da5-a887-f8c9df884fa2-tls-certs\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.340192 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.339917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-tmp-dir\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.340306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.340250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-model-cache\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.340306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.340276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-home\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.340397 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.340337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-tmp-dir\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.342122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.342095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-dshm\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.342507 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.342474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67753ed0-12fe-4da5-a887-f8c9df884fa2-tls-certs\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.347174 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.347148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/67753ed0-12fe-4da5-a887-f8c9df884fa2-kube-api-access-f7qpm\") pod \"distinct-2-w4-7e2add-kserve-646bb8c57-hkkld\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") " pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.391846 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.391818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:22.665872 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.665846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"]
Sep 11 22:15:22.668344 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:15:22.668319    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67753ed0_12fe_4da5_a887_f8c9df884fa2.slice/crio-b38d35dddd714e72df5bc08f19a45df9c9c7c2c6f445acc332af765b5a46d536 WatchSource:0}: Error finding container b38d35dddd714e72df5bc08f19a45df9c9c7c2c6f445acc332af765b5a46d536: Status 404 returned error can't find the container with id b38d35dddd714e72df5bc08f19a45df9c9c7c2c6f445acc332af765b5a46d536
Sep 11 22:15:22.699253 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:22.699224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld" event={"ID":"67753ed0-12fe-4da5-a887-f8c9df884fa2","Type":"ContainerStarted","Data":"b38d35dddd714e72df5bc08f19a45df9c9c7c2c6f445acc332af765b5a46d536"}
Sep 11 22:15:23.707074 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:23.707034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld" event={"ID":"67753ed0-12fe-4da5-a887-f8c9df884fa2","Type":"ContainerStarted","Data":"73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9"}
Sep 11 22:15:23.721671 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:23.721596    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld" podStartSLOduration=1.721578387 podStartE2EDuration="1.721578387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:15:23.721496074 +0000 UTC m=+1385.406442817" watchObservedRunningTime="2026-09-11 22:15:23.721578387 +0000 UTC m=+1385.406525113"
Sep 11 22:15:32.392297 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:32.392246    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:32.392814 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:32.392521    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:32.404961 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:32.404935    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:15:32.778726 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:15:32.778690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:16:19.037930 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:19.037840    2578 scope.go:117] "RemoveContainer" containerID="4cf9b9c074a92aefcf52a7ef82b8b644c361120490dda1e4c6e6b2bc511e5854"
Sep 11 22:16:19.047584 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:19.047560    2578 scope.go:117] "RemoveContainer" containerID="238bb4b9ccdd3cb0e93df71a8aaf229c414f07347d7dedb16affae88f271fb15"
Sep 11 22:16:21.046620 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.046580    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"]
Sep 11 22:16:21.051376 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.051357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:21.053564 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.053538    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fe6196\""
Sep 11 22:16:21.053730 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.053544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fe6196-dockercfg-8tkfc\""
Sep 11 22:16:21.061075 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.061045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"]
Sep 11 22:16:21.150350 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.150305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbpz\" (UniqueName: \"kubernetes.io/projected/61511bc9-3e98-47ec-97f4-466cf4ee9c19-kube-api-access-lhbpz\") pod \"payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg\" (UID: \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:21.150535 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.150455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61511bc9-3e98-47ec-97f4-466cf4ee9c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg\" (UID: \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:21.251600 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.251556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbpz\" (UniqueName: \"kubernetes.io/projected/61511bc9-3e98-47ec-97f4-466cf4ee9c19-kube-api-access-lhbpz\") pod \"payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg\" (UID: \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:21.251813 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.251722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61511bc9-3e98-47ec-97f4-466cf4ee9c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg\" (UID: \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:21.252454 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.252432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61511bc9-3e98-47ec-97f4-466cf4ee9c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg\" (UID: \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:21.258842 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.258817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbpz\" (UniqueName: \"kubernetes.io/projected/61511bc9-3e98-47ec-97f4-466cf4ee9c19-kube-api-access-lhbpz\") pod \"payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg\" (UID: \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:21.364693 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.364575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:21.638740 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:21.638709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"]
Sep 11 22:16:21.639987 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:16:21.639961    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61511bc9_3e98_47ec_97f4_466cf4ee9c19.slice/crio-b7de3f1bb7124276ef48574de0363d0bb5ef201c254a96ca83ed8eb1eadb6e1f WatchSource:0}: Error finding container b7de3f1bb7124276ef48574de0363d0bb5ef201c254a96ca83ed8eb1eadb6e1f: Status 404 returned error can't find the container with id b7de3f1bb7124276ef48574de0363d0bb5ef201c254a96ca83ed8eb1eadb6e1f
Sep 11 22:16:22.075012 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:22.074968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg" event={"ID":"61511bc9-3e98-47ec-97f4-466cf4ee9c19","Type":"ContainerStarted","Data":"b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f"}
Sep 11 22:16:22.075419 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:22.075033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg" event={"ID":"61511bc9-3e98-47ec-97f4-466cf4ee9c19","Type":"ContainerStarted","Data":"b7de3f1bb7124276ef48574de0363d0bb5ef201c254a96ca83ed8eb1eadb6e1f"}
Sep 11 22:16:22.075419 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:22.075064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:22.092718 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:22.092668    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg" podStartSLOduration=1.092653638 podStartE2EDuration="1.092653638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:16:22.089913 +0000 UTC m=+1443.774859715" watchObservedRunningTime="2026-09-11 22:16:22.092653638 +0000 UTC m=+1443.777600343"
Sep 11 22:16:33.084682 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:33.084623    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:16:54.529745 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:54.529709    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"]
Sep 11 22:16:54.533442 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:54.533419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"
Sep 11 22:16:54.535563 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:54.535540    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:16:54.541306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:54.541149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"]
Sep 11 22:16:54.594142 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:54.594098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69898969-5d11-465b-8124-e1fe09fa42f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7\" (UID: \"69898969-5d11-465b-8124-e1fe09fa42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"
Sep 11 22:16:54.695215 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:54.695181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69898969-5d11-465b-8124-e1fe09fa42f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7\" (UID: \"69898969-5d11-465b-8124-e1fe09fa42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"
Sep 11 22:16:54.695861 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:54.695841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69898969-5d11-465b-8124-e1fe09fa42f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7\" (UID: \"69898969-5d11-465b-8124-e1fe09fa42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"
Sep 11 22:16:54.845985 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:54.845886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"
Sep 11 22:16:55.123240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:55.123201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"]
Sep 11 22:16:55.124718 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:16:55.124684    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69898969_5d11_465b_8124_e1fe09fa42f3.slice/crio-f0dfaac32a4f7406160ecfd22339f3894f0cab7bab9c28aab72ba6e66ff9cde5 WatchSource:0}: Error finding container f0dfaac32a4f7406160ecfd22339f3894f0cab7bab9c28aab72ba6e66ff9cde5: Status 404 returned error can't find the container with id f0dfaac32a4f7406160ecfd22339f3894f0cab7bab9c28aab72ba6e66ff9cde5
Sep 11 22:16:55.285242 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:55.285200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7" event={"ID":"69898969-5d11-465b-8124-e1fe09fa42f3","Type":"ContainerStarted","Data":"f0dfaac32a4f7406160ecfd22339f3894f0cab7bab9c28aab72ba6e66ff9cde5"}
Sep 11 22:16:56.294164 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:56.294132    2578 generic.go:356] "Generic (PLEG): container finished" podID="69898969-5d11-465b-8124-e1fe09fa42f3" containerID="d5aed3b9ac09e01a9bebcf993e0ce35be55d4cf2b74b67ecfab2fdc65086fcac" exitCode=0
Sep 11 22:16:56.294164 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:56.294177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7" event={"ID":"69898969-5d11-465b-8124-e1fe09fa42f3","Type":"ContainerDied","Data":"d5aed3b9ac09e01a9bebcf993e0ce35be55d4cf2b74b67ecfab2fdc65086fcac"}
Sep 11 22:16:57.436257 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:57.436230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"
Sep 11 22:16:57.525363 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:57.525318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69898969-5d11-465b-8124-e1fe09fa42f3-maas-api-service-ca\") pod \"69898969-5d11-465b-8124-e1fe09fa42f3\" (UID: \"69898969-5d11-465b-8124-e1fe09fa42f3\") "
Sep 11 22:16:57.525711 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:57.525681    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69898969-5d11-465b-8124-e1fe09fa42f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69898969-5d11-465b-8124-e1fe09fa42f3" (UID: "69898969-5d11-465b-8124-e1fe09fa42f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:16:57.627105 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:57.627015    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69898969-5d11-465b-8124-e1fe09fa42f3-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:16:58.307427 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:58.307394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"
Sep 11 22:16:58.307683 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:58.307430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7" event={"ID":"69898969-5d11-465b-8124-e1fe09fa42f3","Type":"ContainerDied","Data":"f0dfaac32a4f7406160ecfd22339f3894f0cab7bab9c28aab72ba6e66ff9cde5"}
Sep 11 22:16:58.307683 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:58.307487    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0dfaac32a4f7406160ecfd22339f3894f0cab7bab9c28aab72ba6e66ff9cde5"
Sep 11 22:16:59.392253 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:59.392210    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"]
Sep 11 22:16:59.400690 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:59.400660    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"]
Sep 11 22:16:59.429966 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:16:59.429934    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"]
Sep 11 22:17:10.749247 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:10.749197    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"]
Sep 11 22:17:10.752227 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:10.752197    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-95d6c2-w89g7"]
Sep 11 22:17:10.858240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:10.858206    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69898969-5d11-465b-8124-e1fe09fa42f3" path="/var/lib/kubelet/pods/69898969-5d11-465b-8124-e1fe09fa42f3/volumes"
Sep 11 22:17:14.412382 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.412324    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg" podUID="caf2eca8-f317-4e47-9909-e882d5882041" containerName="main" containerID="cri-o://deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c" gracePeriod=45
Sep 11 22:17:14.424063 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.424034    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8" podUID="dbbb5804-1291-49f1-9df9-4496bdc88a42" containerName="main" containerID="cri-o://e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57" gracePeriod=45
Sep 11 22:17:14.449170 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.449136    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87" podUID="f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" containerName="main" containerID="cri-o://ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf" gracePeriod=45
Sep 11 22:17:14.737816 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.737787    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:17:14.742501 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.742481    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:17:14.746149 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.746124    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:17:14.804586 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-dshm\") pod \"dbbb5804-1291-49f1-9df9-4496bdc88a42\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") "
Sep 11 22:17:14.804807 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-home\") pod \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") "
Sep 11 22:17:14.804807 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljbtb\" (UniqueName: \"kubernetes.io/projected/dbbb5804-1291-49f1-9df9-4496bdc88a42-kube-api-access-ljbtb\") pod \"dbbb5804-1291-49f1-9df9-4496bdc88a42\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") "
Sep 11 22:17:14.804807 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbb5804-1291-49f1-9df9-4496bdc88a42-tls-certs\") pod \"dbbb5804-1291-49f1-9df9-4496bdc88a42\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") "
Sep 11 22:17:14.804807 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-home\") pod \"caf2eca8-f317-4e47-9909-e882d5882041\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") "
Sep 11 22:17:14.805030 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804828    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tls-certs\") pod \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") "
Sep 11 22:17:14.805030 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804860    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-home" (OuterVolumeSpecName: "home") pod "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" (UID: "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.805030 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-model-cache\") pod \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") "
Sep 11 22:17:14.805030 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-tmp-dir\") pod \"caf2eca8-f317-4e47-9909-e882d5882041\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") "
Sep 11 22:17:14.805030 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.804994    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpqb9\" (UniqueName: \"kubernetes.io/projected/caf2eca8-f317-4e47-9909-e882d5882041-kube-api-access-tpqb9\") pod \"caf2eca8-f317-4e47-9909-e882d5882041\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") "
Sep 11 22:17:14.805030 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805012    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-home" (OuterVolumeSpecName: "home") pod "caf2eca8-f317-4e47-9909-e882d5882041" (UID: "caf2eca8-f317-4e47-9909-e882d5882041"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.805030 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfl4c\" (UniqueName: \"kubernetes.io/projected/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-kube-api-access-hfl4c\") pod \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") "
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tmp-dir\") pod \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") "
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805102    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-tmp-dir\") pod \"dbbb5804-1291-49f1-9df9-4496bdc88a42\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") "
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-dshm\") pod \"caf2eca8-f317-4e47-9909-e882d5882041\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") "
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-model-cache\") pod \"caf2eca8-f317-4e47-9909-e882d5882041\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") "
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-home\") pod \"dbbb5804-1291-49f1-9df9-4496bdc88a42\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") "
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805211    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" (UID: "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805227    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf2eca8-f317-4e47-9909-e882d5882041-tls-certs\") pod \"caf2eca8-f317-4e47-9909-e882d5882041\" (UID: \"caf2eca8-f317-4e47-9909-e882d5882041\") "
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-model-cache\") pod \"dbbb5804-1291-49f1-9df9-4496bdc88a42\" (UID: \"dbbb5804-1291-49f1-9df9-4496bdc88a42\") "
Sep 11 22:17:14.805378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-dshm\") pod \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\" (UID: \"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0\") "
Sep 11 22:17:14.805856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" (UID: "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.805856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "caf2eca8-f317-4e47-9909-e882d5882041" (UID: "caf2eca8-f317-4e47-9909-e882d5882041"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.805856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805786    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.805856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805849    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.806073 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805866    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.806073 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805879    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.806073 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-model-cache" (OuterVolumeSpecName: "model-cache") pod "caf2eca8-f317-4e47-9909-e882d5882041" (UID: "caf2eca8-f317-4e47-9909-e882d5882041"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.806073 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.805890    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.806284 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.806092    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbbb5804-1291-49f1-9df9-4496bdc88a42" (UID: "dbbb5804-1291-49f1-9df9-4496bdc88a42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.807752 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.807723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-dshm" (OuterVolumeSpecName: "dshm") pod "dbbb5804-1291-49f1-9df9-4496bdc88a42" (UID: "dbbb5804-1291-49f1-9df9-4496bdc88a42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.807951 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.807918    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbbb5804-1291-49f1-9df9-4496bdc88a42" (UID: "dbbb5804-1291-49f1-9df9-4496bdc88a42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.808085 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.808061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-dshm" (OuterVolumeSpecName: "dshm") pod "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" (UID: "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.808273 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.808251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-home" (OuterVolumeSpecName: "home") pod "dbbb5804-1291-49f1-9df9-4496bdc88a42" (UID: "dbbb5804-1291-49f1-9df9-4496bdc88a42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.808409 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.808383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" (UID: "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:14.808476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.808452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbbb5804-1291-49f1-9df9-4496bdc88a42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbbb5804-1291-49f1-9df9-4496bdc88a42" (UID: "dbbb5804-1291-49f1-9df9-4496bdc88a42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:14.808656 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.808613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf2eca8-f317-4e47-9909-e882d5882041-kube-api-access-tpqb9" (OuterVolumeSpecName: "kube-api-access-tpqb9") pod "caf2eca8-f317-4e47-9909-e882d5882041" (UID: "caf2eca8-f317-4e47-9909-e882d5882041"). InnerVolumeSpecName "kube-api-access-tpqb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:14.809217 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.809190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-dshm" (OuterVolumeSpecName: "dshm") pod "caf2eca8-f317-4e47-9909-e882d5882041" (UID: "caf2eca8-f317-4e47-9909-e882d5882041"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:14.809821 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.809792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbb5804-1291-49f1-9df9-4496bdc88a42-kube-api-access-ljbtb" (OuterVolumeSpecName: "kube-api-access-ljbtb") pod "dbbb5804-1291-49f1-9df9-4496bdc88a42" (UID: "dbbb5804-1291-49f1-9df9-4496bdc88a42"). InnerVolumeSpecName "kube-api-access-ljbtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:14.809964 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.809936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf2eca8-f317-4e47-9909-e882d5882041-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "caf2eca8-f317-4e47-9909-e882d5882041" (UID: "caf2eca8-f317-4e47-9909-e882d5882041"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:14.810200 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.810178    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-kube-api-access-hfl4c" (OuterVolumeSpecName: "kube-api-access-hfl4c") pod "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" (UID: "f4b84bf8-1509-4eb1-81ea-475d1a1f78a0"). InnerVolumeSpecName "kube-api-access-hfl4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:14.906406 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906374    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfl4c\" (UniqueName: \"kubernetes.io/projected/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-kube-api-access-hfl4c\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906406 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906401    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906406 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906411    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906419    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf2eca8-f317-4e47-9909-e882d5882041-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906436    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906444    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf2eca8-f317-4e47-9909-e882d5882041-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906452    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906459    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906466    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbb5804-1291-49f1-9df9-4496bdc88a42-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906473    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljbtb\" (UniqueName: \"kubernetes.io/projected/dbbb5804-1291-49f1-9df9-4496bdc88a42-kube-api-access-ljbtb\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906481    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbb5804-1291-49f1-9df9-4496bdc88a42-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906488    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:14.906643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:14.906496    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpqb9\" (UniqueName: \"kubernetes.io/projected/caf2eca8-f317-4e47-9909-e882d5882041-kube-api-access-tpqb9\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:15.412200 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.412153    2578 generic.go:356] "Generic (PLEG): container finished" podID="caf2eca8-f317-4e47-9909-e882d5882041" containerID="deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c" exitCode=0
Sep 11 22:17:15.412393 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.412211    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"
Sep 11 22:17:15.412393 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.412236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg" event={"ID":"caf2eca8-f317-4e47-9909-e882d5882041","Type":"ContainerDied","Data":"deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c"}
Sep 11 22:17:15.412393 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.412278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg" event={"ID":"caf2eca8-f317-4e47-9909-e882d5882041","Type":"ContainerDied","Data":"0057520ff107d78f0fa1426d3943e2ef52d0e9f96332fe957c78a5f73c79584f"}
Sep 11 22:17:15.412393 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.412294    2578 scope.go:117] "RemoveContainer" containerID="deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c"
Sep 11 22:17:15.413859 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.413830    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" containerID="ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf" exitCode=0
Sep 11 22:17:15.413948 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.413879    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"
Sep 11 22:17:15.413948 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.413906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87" event={"ID":"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0","Type":"ContainerDied","Data":"ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf"}
Sep 11 22:17:15.413948 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.413944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87" event={"ID":"f4b84bf8-1509-4eb1-81ea-475d1a1f78a0","Type":"ContainerDied","Data":"d38518574f0e5e9a75a10d267400c4ce8658490d879a48559be38aebbac5a418"}
Sep 11 22:17:15.415864 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.415839    2578 generic.go:356] "Generic (PLEG): container finished" podID="dbbb5804-1291-49f1-9df9-4496bdc88a42" containerID="e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57" exitCode=0
Sep 11 22:17:15.415960 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.415885    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"
Sep 11 22:17:15.415960 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.415929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8" event={"ID":"dbbb5804-1291-49f1-9df9-4496bdc88a42","Type":"ContainerDied","Data":"e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57"}
Sep 11 22:17:15.416107 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.415977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8" event={"ID":"dbbb5804-1291-49f1-9df9-4496bdc88a42","Type":"ContainerDied","Data":"5cba850bb8ff77dfdb06e94ef7f59148ccca44d9430f19149a2667b343b5dee3"}
Sep 11 22:17:15.423753 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.423730    2578 scope.go:117] "RemoveContainer" containerID="deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c"
Sep 11 22:17:15.424101 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:17:15.424080    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c\": container with ID starting with deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c not found: ID does not exist" containerID="deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c"
Sep 11 22:17:15.424155 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.424109    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c"} err="failed to get container status \"deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c\": rpc error: code = NotFound desc = could not find container \"deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c\": container with ID starting with deedcbfe3abd16b4c412a1364322ec00a42da85aac172bbb158581bcf6b49b6c not found: ID does not exist"
Sep 11 22:17:15.424155 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.424127    2578 scope.go:117] "RemoveContainer" containerID="ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf"
Sep 11 22:17:15.433945 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.433915    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"]
Sep 11 22:17:15.437822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.437797    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-2-w0-95d6c2-kserve-d995c55b5-wztdg"]
Sep 11 22:17:15.441873 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.441855    2578 scope.go:117] "RemoveContainer" containerID="ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf"
Sep 11 22:17:15.442190 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:17:15.442175    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf\": container with ID starting with ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf not found: ID does not exist" containerID="ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf"
Sep 11 22:17:15.442242 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.442199    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf"} err="failed to get container status \"ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf\": rpc error: code = NotFound desc = could not find container \"ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf\": container with ID starting with ea55c16bb4863743214ebfd6be53278327650a83371caa2a402c9cbd46ae2ecf not found: ID does not exist"
Sep 11 22:17:15.442242 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.442218    2578 scope.go:117] "RemoveContainer" containerID="e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57"
Sep 11 22:17:15.447397 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.447370    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"]
Sep 11 22:17:15.451508 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.451483    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/distinct-w0-95d6c2-kserve-65647494bb-8fqv8"]
Sep 11 22:17:15.452000 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.451983    2578 scope.go:117] "RemoveContainer" containerID="e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57"
Sep 11 22:17:15.452753 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:17:15.452457    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57\": container with ID starting with e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57 not found: ID does not exist" containerID="e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57"
Sep 11 22:17:15.452753 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.452492    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57"} err="failed to get container status \"e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57\": rpc error: code = NotFound desc = could not find container \"e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57\": container with ID starting with e20707605607733ba8ce69f0c04fb7a13976226f0c1db53919f358538084ea57 not found: ID does not exist"
Sep 11 22:17:15.459842 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.459818    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"]
Sep 11 22:17:15.465152 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:15.465127    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-95d6c2/simulator-w0-95d6c2-kserve-5b9fc9586b-wbl87"]
Sep 11 22:17:16.856887 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:16.856842    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf2eca8-f317-4e47-9909-e882d5882041" path="/var/lib/kubelet/pods/caf2eca8-f317-4e47-9909-e882d5882041/volumes"
Sep 11 22:17:16.857465 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:16.857442    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbbb5804-1291-49f1-9df9-4496bdc88a42" path="/var/lib/kubelet/pods/dbbb5804-1291-49f1-9df9-4496bdc88a42/volumes"
Sep 11 22:17:16.857904 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:16.857883    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" path="/var/lib/kubelet/pods/f4b84bf8-1509-4eb1-81ea-475d1a1f78a0/volumes"
Sep 11 22:17:19.076339 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:19.076307    2578 scope.go:117] "RemoveContainer" containerID="00e2195419a98b9d082daa575aab309807516d956e81a1e4afea9463d1fae256"
Sep 11 22:17:19.085363 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:19.085341    2578 scope.go:117] "RemoveContainer" containerID="2ee60ba0d9637d34f130d5f07a5dfabe9827a140f68bc96cac22d20f7481fc64"
Sep 11 22:17:22.385841 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.385807    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"]
Sep 11 22:17:22.386338 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386322    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbbb5804-1291-49f1-9df9-4496bdc88a42" containerName="main"
Sep 11 22:17:22.386398 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386340    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbbb5804-1291-49f1-9df9-4496bdc88a42" containerName="main"
Sep 11 22:17:22.386398 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf2eca8-f317-4e47-9909-e882d5882041" containerName="main"
Sep 11 22:17:22.386398 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386362    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf2eca8-f317-4e47-9909-e882d5882041" containerName="main"
Sep 11 22:17:22.386398 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386371    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" containerName="main"
Sep 11 22:17:22.386398 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" containerName="main"
Sep 11 22:17:22.386398 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386388    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69898969-5d11-465b-8124-e1fe09fa42f3" containerName="revoke-keys"
Sep 11 22:17:22.386398 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386393    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="69898969-5d11-465b-8124-e1fe09fa42f3" containerName="revoke-keys"
Sep 11 22:17:22.386623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386464    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbbb5804-1291-49f1-9df9-4496bdc88a42" containerName="main"
Sep 11 22:17:22.386623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386475    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b84bf8-1509-4eb1-81ea-475d1a1f78a0" containerName="main"
Sep 11 22:17:22.386623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386482    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="69898969-5d11-465b-8124-e1fe09fa42f3" containerName="revoke-keys"
Sep 11 22:17:22.386623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.386489    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf2eca8-f317-4e47-9909-e882d5882041" containerName="main"
Sep 11 22:17:22.389512 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.389495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"
Sep 11 22:17:22.391749 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.391725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:17:22.399574 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.399542    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"]
Sep 11 22:17:22.486819 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.486785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82818d5a-706e-4309-9f81-c6247d9a7f81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe6196-59q8t\" (UID: \"82818d5a-706e-4309-9f81-c6247d9a7f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"
Sep 11 22:17:22.587984 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.587948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82818d5a-706e-4309-9f81-c6247d9a7f81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe6196-59q8t\" (UID: \"82818d5a-706e-4309-9f81-c6247d9a7f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"
Sep 11 22:17:22.588614 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.588591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82818d5a-706e-4309-9f81-c6247d9a7f81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe6196-59q8t\" (UID: \"82818d5a-706e-4309-9f81-c6247d9a7f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"
Sep 11 22:17:22.708463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.708423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"
Sep 11 22:17:22.980444 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:22.980416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"]
Sep 11 22:17:22.981950 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:17:22.981918    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82818d5a_706e_4309_9f81_c6247d9a7f81.slice/crio-67d79d7a5e343b2351f24769d2657e9bf2d569c765a7dae7abb7e0f57addd0fd WatchSource:0}: Error finding container 67d79d7a5e343b2351f24769d2657e9bf2d569c765a7dae7abb7e0f57addd0fd: Status 404 returned error can't find the container with id 67d79d7a5e343b2351f24769d2657e9bf2d569c765a7dae7abb7e0f57addd0fd
Sep 11 22:17:23.469238 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:23.469189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t" event={"ID":"82818d5a-706e-4309-9f81-c6247d9a7f81","Type":"ContainerStarted","Data":"f55f81d847c8cbb3ad2d23e4f103f8f5550b1ff9491cef7d321d26b746b478fa"}
Sep 11 22:17:23.469678 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:23.469257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t" event={"ID":"82818d5a-706e-4309-9f81-c6247d9a7f81","Type":"ContainerStarted","Data":"67d79d7a5e343b2351f24769d2657e9bf2d569c765a7dae7abb7e0f57addd0fd"}
Sep 11 22:17:23.484298 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:23.484218    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t" podStartSLOduration=1.4841955869999999 podStartE2EDuration="1.484195587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:17:23.483783438 +0000 UTC m=+1505.168730148" watchObservedRunningTime="2026-09-11 22:17:23.484195587 +0000 UTC m=+1505.169142302"
Sep 11 22:17:24.476164 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:24.476124    2578 generic.go:356] "Generic (PLEG): container finished" podID="82818d5a-706e-4309-9f81-c6247d9a7f81" containerID="f55f81d847c8cbb3ad2d23e4f103f8f5550b1ff9491cef7d321d26b746b478fa" exitCode=0
Sep 11 22:17:24.476601 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:24.476208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t" event={"ID":"82818d5a-706e-4309-9f81-c6247d9a7f81","Type":"ContainerDied","Data":"f55f81d847c8cbb3ad2d23e4f103f8f5550b1ff9491cef7d321d26b746b478fa"}
Sep 11 22:17:25.614889 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:25.614863    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"
Sep 11 22:17:25.717592 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:25.717555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82818d5a-706e-4309-9f81-c6247d9a7f81-maas-api-service-ca\") pod \"82818d5a-706e-4309-9f81-c6247d9a7f81\" (UID: \"82818d5a-706e-4309-9f81-c6247d9a7f81\") "
Sep 11 22:17:25.717970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:25.717942    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82818d5a-706e-4309-9f81-c6247d9a7f81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82818d5a-706e-4309-9f81-c6247d9a7f81" (UID: "82818d5a-706e-4309-9f81-c6247d9a7f81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:25.718163 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:25.718139    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82818d5a-706e-4309-9f81-c6247d9a7f81-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:26.489560 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:26.489529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"
Sep 11 22:17:26.489560 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:26.489545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t" event={"ID":"82818d5a-706e-4309-9f81-c6247d9a7f81","Type":"ContainerDied","Data":"67d79d7a5e343b2351f24769d2657e9bf2d569c765a7dae7abb7e0f57addd0fd"}
Sep 11 22:17:26.489785 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:26.489593    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d79d7a5e343b2351f24769d2657e9bf2d569c765a7dae7abb7e0f57addd0fd"
Sep 11 22:17:38.238228 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.238189    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"]
Sep 11 22:17:38.238745 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.238415    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg" podUID="61511bc9-3e98-47ec-97f4-466cf4ee9c19" containerName="payload-pre-processing" containerID="cri-o://b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f" gracePeriod=30
Sep 11 22:17:38.400162 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.400083    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"]
Sep 11 22:17:38.403335 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.403304    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe6196-59q8t"]
Sep 11 22:17:38.503732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.503673    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:17:38.566865 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.566829    2578 generic.go:356] "Generic (PLEG): container finished" podID="61511bc9-3e98-47ec-97f4-466cf4ee9c19" containerID="b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f" exitCode=0
Sep 11 22:17:38.567039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.566875    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"
Sep 11 22:17:38.567039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.566921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg" event={"ID":"61511bc9-3e98-47ec-97f4-466cf4ee9c19","Type":"ContainerDied","Data":"b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f"}
Sep 11 22:17:38.567039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.566998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg" event={"ID":"61511bc9-3e98-47ec-97f4-466cf4ee9c19","Type":"ContainerDied","Data":"b7de3f1bb7124276ef48574de0363d0bb5ef201c254a96ca83ed8eb1eadb6e1f"}
Sep 11 22:17:38.567039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.567021    2578 scope.go:117] "RemoveContainer" containerID="b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f"
Sep 11 22:17:38.577231 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.577207    2578 scope.go:117] "RemoveContainer" containerID="b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f"
Sep 11 22:17:38.577529 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:17:38.577510    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f\": container with ID starting with b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f not found: ID does not exist" containerID="b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f"
Sep 11 22:17:38.577612 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.577537    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f"} err="failed to get container status \"b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f\": rpc error: code = NotFound desc = could not find container \"b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f\": container with ID starting with b4486e4b4663e76ef412bb08160eb25cfc14bd2eabb64f4b079bccd57e371e7f not found: ID does not exist"
Sep 11 22:17:38.649240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.649208    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhbpz\" (UniqueName: \"kubernetes.io/projected/61511bc9-3e98-47ec-97f4-466cf4ee9c19-kube-api-access-lhbpz\") pod \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\" (UID: \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\") "
Sep 11 22:17:38.649419 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.649259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61511bc9-3e98-47ec-97f4-466cf4ee9c19-plugins-config-volume\") pod \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\" (UID: \"61511bc9-3e98-47ec-97f4-466cf4ee9c19\") "
Sep 11 22:17:38.649728 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.649704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61511bc9-3e98-47ec-97f4-466cf4ee9c19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61511bc9-3e98-47ec-97f4-466cf4ee9c19" (UID: "61511bc9-3e98-47ec-97f4-466cf4ee9c19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:38.651382 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.651363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61511bc9-3e98-47ec-97f4-466cf4ee9c19-kube-api-access-lhbpz" (OuterVolumeSpecName: "kube-api-access-lhbpz") pod "61511bc9-3e98-47ec-97f4-466cf4ee9c19" (UID: "61511bc9-3e98-47ec-97f4-466cf4ee9c19"). InnerVolumeSpecName "kube-api-access-lhbpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:38.750199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.750152    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhbpz\" (UniqueName: \"kubernetes.io/projected/61511bc9-3e98-47ec-97f4-466cf4ee9c19-kube-api-access-lhbpz\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:38.750199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.750193    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61511bc9-3e98-47ec-97f4-466cf4ee9c19-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:38.857887 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.857810    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82818d5a-706e-4309-9f81-c6247d9a7f81" path="/var/lib/kubelet/pods/82818d5a-706e-4309-9f81-c6247d9a7f81/volumes"
Sep 11 22:17:38.881045 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.881010    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"]
Sep 11 22:17:38.884976 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:38.884949    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe6196-84849bfbfd-p49cg"]
Sep 11 22:17:40.857093 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:40.857060    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61511bc9-3e98-47ec-97f4-466cf4ee9c19" path="/var/lib/kubelet/pods/61511bc9-3e98-47ec-97f4-466cf4ee9c19/volumes"
Sep 11 22:17:45.196500 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.196463    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"]
Sep 11 22:17:45.196991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.196861    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82818d5a-706e-4309-9f81-c6247d9a7f81" containerName="revoke-keys"
Sep 11 22:17:45.196991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.196873    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="82818d5a-706e-4309-9f81-c6247d9a7f81" containerName="revoke-keys"
Sep 11 22:17:45.196991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.196892    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61511bc9-3e98-47ec-97f4-466cf4ee9c19" containerName="payload-pre-processing"
Sep 11 22:17:45.196991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.196898    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61511bc9-3e98-47ec-97f4-466cf4ee9c19" containerName="payload-pre-processing"
Sep 11 22:17:45.196991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.196960    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61511bc9-3e98-47ec-97f4-466cf4ee9c19" containerName="payload-pre-processing"
Sep 11 22:17:45.196991 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.196971    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="82818d5a-706e-4309-9f81-c6247d9a7f81" containerName="revoke-keys"
Sep 11 22:17:45.201422 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.201405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"
Sep 11 22:17:45.203687 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.203662    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:17:45.206205 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.206181    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"]
Sep 11 22:17:45.213002 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.212977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c2ebe6-e194-474d-9280-c8818147fffe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj\" (UID: \"23c2ebe6-e194-474d-9280-c8818147fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"
Sep 11 22:17:45.314140 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.314103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c2ebe6-e194-474d-9280-c8818147fffe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj\" (UID: \"23c2ebe6-e194-474d-9280-c8818147fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"
Sep 11 22:17:45.315020 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.314995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c2ebe6-e194-474d-9280-c8818147fffe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj\" (UID: \"23c2ebe6-e194-474d-9280-c8818147fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"
Sep 11 22:17:45.514394 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.514311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"
Sep 11 22:17:45.784643 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:45.784598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"]
Sep 11 22:17:45.786124 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:17:45.786096    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c2ebe6_e194_474d_9280_c8818147fffe.slice/crio-d6d0191a76bcf654dc1831a72f513f0d6ccc2e9ed53ce9845fa6c225b821b377 WatchSource:0}: Error finding container d6d0191a76bcf654dc1831a72f513f0d6ccc2e9ed53ce9845fa6c225b821b377: Status 404 returned error can't find the container with id d6d0191a76bcf654dc1831a72f513f0d6ccc2e9ed53ce9845fa6c225b821b377
Sep 11 22:17:46.616173 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:46.616130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj" event={"ID":"23c2ebe6-e194-474d-9280-c8818147fffe","Type":"ContainerStarted","Data":"fe81b2efc5b100a961db62a0a8f3f5174c74aab328beb4fb33df1dd39e647b06"}
Sep 11 22:17:46.616561 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:46.616193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj" event={"ID":"23c2ebe6-e194-474d-9280-c8818147fffe","Type":"ContainerStarted","Data":"d6d0191a76bcf654dc1831a72f513f0d6ccc2e9ed53ce9845fa6c225b821b377"}
Sep 11 22:17:46.630851 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:46.630795    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj" podStartSLOduration=1.63077487 podStartE2EDuration="1.63077487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:17:46.629143266 +0000 UTC m=+1528.314089981" watchObservedRunningTime="2026-09-11 22:17:46.63077487 +0000 UTC m=+1528.315721585"
Sep 11 22:17:47.623527 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:47.623493    2578 generic.go:356] "Generic (PLEG): container finished" podID="23c2ebe6-e194-474d-9280-c8818147fffe" containerID="fe81b2efc5b100a961db62a0a8f3f5174c74aab328beb4fb33df1dd39e647b06" exitCode=0
Sep 11 22:17:47.623929 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:47.623547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj" event={"ID":"23c2ebe6-e194-474d-9280-c8818147fffe","Type":"ContainerDied","Data":"fe81b2efc5b100a961db62a0a8f3f5174c74aab328beb4fb33df1dd39e647b06"}
Sep 11 22:17:48.765693 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:48.765664    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"
Sep 11 22:17:48.859401 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:48.859370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c2ebe6-e194-474d-9280-c8818147fffe-maas-api-service-ca\") pod \"23c2ebe6-e194-474d-9280-c8818147fffe\" (UID: \"23c2ebe6-e194-474d-9280-c8818147fffe\") "
Sep 11 22:17:48.859736 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:48.859714    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c2ebe6-e194-474d-9280-c8818147fffe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23c2ebe6-e194-474d-9280-c8818147fffe" (UID: "23c2ebe6-e194-474d-9280-c8818147fffe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:48.960076 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:48.960043    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c2ebe6-e194-474d-9280-c8818147fffe-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:49.638049 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:49.638017    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"
Sep 11 22:17:49.638203 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:49.638021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj" event={"ID":"23c2ebe6-e194-474d-9280-c8818147fffe","Type":"ContainerDied","Data":"d6d0191a76bcf654dc1831a72f513f0d6ccc2e9ed53ce9845fa6c225b821b377"}
Sep 11 22:17:49.638203 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:49.638139    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6d0191a76bcf654dc1831a72f513f0d6ccc2e9ed53ce9845fa6c225b821b377"
Sep 11 22:17:53.348056 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.348019    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"]
Sep 11 22:17:53.348424 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.348409    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c2ebe6-e194-474d-9280-c8818147fffe" containerName="revoke-keys"
Sep 11 22:17:53.348424 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.348421    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c2ebe6-e194-474d-9280-c8818147fffe" containerName="revoke-keys"
Sep 11 22:17:53.348517 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.348505    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c2ebe6-e194-474d-9280-c8818147fffe" containerName="revoke-keys"
Sep 11 22:17:53.351617 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.351597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"
Sep 11 22:17:53.353665 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.353622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:17:53.360378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.360345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"]
Sep 11 22:17:53.406593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.406549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e82c92d-9990-4749-928e-01aa986f7b18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh\" (UID: \"9e82c92d-9990-4749-928e-01aa986f7b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"
Sep 11 22:17:53.508140 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.508097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e82c92d-9990-4749-928e-01aa986f7b18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh\" (UID: \"9e82c92d-9990-4749-928e-01aa986f7b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"
Sep 11 22:17:53.508800 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.508780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e82c92d-9990-4749-928e-01aa986f7b18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh\" (UID: \"9e82c92d-9990-4749-928e-01aa986f7b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"
Sep 11 22:17:53.663534 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.663434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"
Sep 11 22:17:53.940002 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:53.939975    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"]
Sep 11 22:17:53.941862 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:17:53.941830    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e82c92d_9990_4749_928e_01aa986f7b18.slice/crio-3ed147b9a1dee9316203ac6a930e4bc318a2fd9495e06c0adb7e8b42df37faa1 WatchSource:0}: Error finding container 3ed147b9a1dee9316203ac6a930e4bc318a2fd9495e06c0adb7e8b42df37faa1: Status 404 returned error can't find the container with id 3ed147b9a1dee9316203ac6a930e4bc318a2fd9495e06c0adb7e8b42df37faa1
Sep 11 22:17:54.671342 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:54.671309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh" event={"ID":"9e82c92d-9990-4749-928e-01aa986f7b18","Type":"ContainerStarted","Data":"b51040932a715847b3744f7f7020b777870e6cbf77cb99dc6368a74ccb60b1d7"}
Sep 11 22:17:54.671723 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:54.671364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh" event={"ID":"9e82c92d-9990-4749-928e-01aa986f7b18","Type":"ContainerStarted","Data":"3ed147b9a1dee9316203ac6a930e4bc318a2fd9495e06c0adb7e8b42df37faa1"}
Sep 11 22:17:54.686390 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:54.686333    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh" podStartSLOduration=1.6863171540000002 podStartE2EDuration="1.686317154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:17:54.683970352 +0000 UTC m=+1536.368917069" watchObservedRunningTime="2026-09-11 22:17:54.686317154 +0000 UTC m=+1536.371263869"
Sep 11 22:17:55.678398 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:55.678362    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e82c92d-9990-4749-928e-01aa986f7b18" containerID="b51040932a715847b3744f7f7020b777870e6cbf77cb99dc6368a74ccb60b1d7" exitCode=0
Sep 11 22:17:55.680763 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:55.678447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh" event={"ID":"9e82c92d-9990-4749-928e-01aa986f7b18","Type":"ContainerDied","Data":"b51040932a715847b3744f7f7020b777870e6cbf77cb99dc6368a74ccb60b1d7"}
Sep 11 22:17:56.817044 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:56.817018    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"
Sep 11 22:17:56.942397 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:56.942289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e82c92d-9990-4749-928e-01aa986f7b18-maas-api-service-ca\") pod \"9e82c92d-9990-4749-928e-01aa986f7b18\" (UID: \"9e82c92d-9990-4749-928e-01aa986f7b18\") "
Sep 11 22:17:56.942870 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:56.942841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e82c92d-9990-4749-928e-01aa986f7b18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e82c92d-9990-4749-928e-01aa986f7b18" (UID: "9e82c92d-9990-4749-928e-01aa986f7b18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:57.043512 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:57.043474    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e82c92d-9990-4749-928e-01aa986f7b18-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:17:57.692596 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:57.692561    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"
Sep 11 22:17:57.692815 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:57.692593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh" event={"ID":"9e82c92d-9990-4749-928e-01aa986f7b18","Type":"ContainerDied","Data":"3ed147b9a1dee9316203ac6a930e4bc318a2fd9495e06c0adb7e8b42df37faa1"}
Sep 11 22:17:57.692815 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:57.692656    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ed147b9a1dee9316203ac6a930e4bc318a2fd9495e06c0adb7e8b42df37faa1"
Sep 11 22:17:58.219556 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:58.219515    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"]
Sep 11 22:17:58.227915 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:58.227328    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"]
Sep 11 22:17:58.242139 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:17:58.242107    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"]
Sep 11 22:18:04.703768 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:04.703730    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"]
Sep 11 22:18:04.706464 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:04.706435    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2715d7-nx6wj"]
Sep 11 22:18:04.857678 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:04.857622    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c2ebe6-e194-474d-9280-c8818147fffe" path="/var/lib/kubelet/pods/23c2ebe6-e194-474d-9280-c8818147fffe/volumes"
Sep 11 22:18:05.993871 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:05.993836    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"]
Sep 11 22:18:05.996264 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:05.996243    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7e2add-29819415-pfzt4"]
Sep 11 22:18:06.252528 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:06.252442    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"]
Sep 11 22:18:06.256786 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:06.256761    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7e2add-smnmh"]
Sep 11 22:18:06.857711 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:06.857678    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc893eb-06ce-4ac8-af38-adf8817ace84" path="/var/lib/kubelet/pods/2bc893eb-06ce-4ac8-af38-adf8817ace84/volumes"
Sep 11 22:18:06.858031 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:06.858017    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e82c92d-9990-4749-928e-01aa986f7b18" path="/var/lib/kubelet/pods/9e82c92d-9990-4749-928e-01aa986f7b18/volumes"
Sep 11 22:18:11.643535 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.643497    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"]
Sep 11 22:18:11.644156 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.644135    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e82c92d-9990-4749-928e-01aa986f7b18" containerName="revoke-keys"
Sep 11 22:18:11.644242 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.644159    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e82c92d-9990-4749-928e-01aa986f7b18" containerName="revoke-keys"
Sep 11 22:18:11.644299 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.644257    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e82c92d-9990-4749-928e-01aa986f7b18" containerName="revoke-keys"
Sep 11 22:18:11.648385 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.648365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"
Sep 11 22:18:11.650889 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.650861    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:18:11.665699 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.665674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"]
Sep 11 22:18:11.787093 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.787057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1587f9a4-b86f-4c57-afbd-7330269c26c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc\" (UID: \"1587f9a4-b86f-4c57-afbd-7330269c26c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"
Sep 11 22:18:11.888890 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.888846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1587f9a4-b86f-4c57-afbd-7330269c26c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc\" (UID: \"1587f9a4-b86f-4c57-afbd-7330269c26c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"
Sep 11 22:18:11.889679 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.889657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1587f9a4-b86f-4c57-afbd-7330269c26c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc\" (UID: \"1587f9a4-b86f-4c57-afbd-7330269c26c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"
Sep 11 22:18:11.958520 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:11.958490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"
Sep 11 22:18:12.235428 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:12.235396    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"]
Sep 11 22:18:12.237602 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:18:12.237574    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1587f9a4_b86f_4c57_afbd_7330269c26c3.slice/crio-fdea98d0bf4295fc1da8d26d2894b92e836230154bb40eb001f31f7fcdffe3dc WatchSource:0}: Error finding container fdea98d0bf4295fc1da8d26d2894b92e836230154bb40eb001f31f7fcdffe3dc: Status 404 returned error can't find the container with id fdea98d0bf4295fc1da8d26d2894b92e836230154bb40eb001f31f7fcdffe3dc
Sep 11 22:18:12.652487 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:12.652453    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"]
Sep 11 22:18:12.652964 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:12.652683    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy" containerID="cri-o://eff66ff9177e6c97dd3a8eb2fc0fd6f136d00d1d817a40682d2915fae1c098e3" gracePeriod=30
Sep 11 22:18:12.786035 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:12.785930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc" event={"ID":"1587f9a4-b86f-4c57-afbd-7330269c26c3","Type":"ContainerStarted","Data":"e1f9ff0086963b1e4c752fe4e746f0fc886b40eeeaf1875f821924253e20a3c9"}
Sep 11 22:18:12.786035 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:12.785994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc" event={"ID":"1587f9a4-b86f-4c57-afbd-7330269c26c3","Type":"ContainerStarted","Data":"fdea98d0bf4295fc1da8d26d2894b92e836230154bb40eb001f31f7fcdffe3dc"}
Sep 11 22:18:12.805248 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:12.805200    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc" podStartSLOduration=1.805185722 podStartE2EDuration="1.805185722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:18:12.802572872 +0000 UTC m=+1554.487519604" watchObservedRunningTime="2026-09-11 22:18:12.805185722 +0000 UTC m=+1554.490132435"
Sep 11 22:18:13.243646 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.243591    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld" podUID="67753ed0-12fe-4da5-a887-f8c9df884fa2" containerName="main" containerID="cri-o://73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9" gracePeriod=45
Sep 11 22:18:13.248202 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.248167    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m" podUID="86b9512f-6a84-439e-8ebd-51d8d18f5e88" containerName="main" containerID="cri-o://a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d" gracePeriod=45
Sep 11 22:18:13.261370 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.261341    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k" podUID="406dcdd6-0d83-4635-8155-acde6c4d46a0" containerName="main" containerID="cri-o://d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3" gracePeriod=45
Sep 11 22:18:13.377946 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.377835    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m" podUID="86b9512f-6a84-439e-8ebd-51d8d18f5e88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 11 22:18:13.557489 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.557460    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:18:13.561201 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.561166    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:18:13.566009 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.565987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:18:13.705345 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705312    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tls-certs\") pod \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") "
Sep 11 22:18:13.705345 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-dshm\") pod \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") "
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-268kf\" (UniqueName: \"kubernetes.io/projected/86b9512f-6a84-439e-8ebd-51d8d18f5e88-kube-api-access-268kf\") pod \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") "
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-tmp-dir\") pod \"406dcdd6-0d83-4635-8155-acde6c4d46a0\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") "
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406dcdd6-0d83-4635-8155-acde6c4d46a0-tls-certs\") pod \"406dcdd6-0d83-4635-8155-acde6c4d46a0\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") "
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tmp-dir\") pod \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") "
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705491    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67753ed0-12fe-4da5-a887-f8c9df884fa2-tls-certs\") pod \"67753ed0-12fe-4da5-a887-f8c9df884fa2\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") "
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-model-cache\") pod \"406dcdd6-0d83-4635-8155-acde6c4d46a0\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") "
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "406dcdd6-0d83-4635-8155-acde6c4d46a0" (UID: "406dcdd6-0d83-4635-8155-acde6c4d46a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-home\") pod \"67753ed0-12fe-4da5-a887-f8c9df884fa2\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") "
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705809    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86b9512f-6a84-439e-8ebd-51d8d18f5e88" (UID: "86b9512f-6a84-439e-8ebd-51d8d18f5e88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.705881 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705842    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "406dcdd6-0d83-4635-8155-acde6c4d46a0" (UID: "406dcdd6-0d83-4635-8155-acde6c4d46a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.706365 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.705916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-dshm\") pod \"406dcdd6-0d83-4635-8155-acde6c4d46a0\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") "
Sep 11 22:18:13.706365 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-home\") pod \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") "
Sep 11 22:18:13.706365 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706111    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-home" (OuterVolumeSpecName: "home") pod "67753ed0-12fe-4da5-a887-f8c9df884fa2" (UID: "67753ed0-12fe-4da5-a887-f8c9df884fa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.706365 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706351    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-home\") pod \"406dcdd6-0d83-4635-8155-acde6c4d46a0\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") "
Sep 11 22:18:13.706609 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-dshm\") pod \"67753ed0-12fe-4da5-a887-f8c9df884fa2\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") "
Sep 11 22:18:13.706609 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkptf\" (UniqueName: \"kubernetes.io/projected/406dcdd6-0d83-4635-8155-acde6c4d46a0-kube-api-access-zkptf\") pod \"406dcdd6-0d83-4635-8155-acde6c4d46a0\" (UID: \"406dcdd6-0d83-4635-8155-acde6c4d46a0\") "
Sep 11 22:18:13.706609 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/67753ed0-12fe-4da5-a887-f8c9df884fa2-kube-api-access-f7qpm\") pod \"67753ed0-12fe-4da5-a887-f8c9df884fa2\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") "
Sep 11 22:18:13.706609 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-home" (OuterVolumeSpecName: "home") pod "86b9512f-6a84-439e-8ebd-51d8d18f5e88" (UID: "86b9512f-6a84-439e-8ebd-51d8d18f5e88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.706609 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-tmp-dir\") pod \"67753ed0-12fe-4da5-a887-f8c9df884fa2\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") "
Sep 11 22:18:13.706906 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-model-cache\") pod \"67753ed0-12fe-4da5-a887-f8c9df884fa2\" (UID: \"67753ed0-12fe-4da5-a887-f8c9df884fa2\") "
Sep 11 22:18:13.706906 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.706692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-model-cache\") pod \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\" (UID: \"86b9512f-6a84-439e-8ebd-51d8d18f5e88\") "
Sep 11 22:18:13.707151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.707130    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.707230 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.707156    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.707230 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.707173    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.707230 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.707195    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.707230 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.707208    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.707447 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.707423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-model-cache" (OuterVolumeSpecName: "model-cache") pod "86b9512f-6a84-439e-8ebd-51d8d18f5e88" (UID: "86b9512f-6a84-439e-8ebd-51d8d18f5e88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.708136 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-dshm" (OuterVolumeSpecName: "dshm") pod "86b9512f-6a84-439e-8ebd-51d8d18f5e88" (UID: "86b9512f-6a84-439e-8ebd-51d8d18f5e88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.708265 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67753ed0-12fe-4da5-a887-f8c9df884fa2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67753ed0-12fe-4da5-a887-f8c9df884fa2" (UID: "67753ed0-12fe-4da5-a887-f8c9df884fa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:13.708265 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708203    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86b9512f-6a84-439e-8ebd-51d8d18f5e88" (UID: "86b9512f-6a84-439e-8ebd-51d8d18f5e88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:13.708387 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708344    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-home" (OuterVolumeSpecName: "home") pod "406dcdd6-0d83-4635-8155-acde6c4d46a0" (UID: "406dcdd6-0d83-4635-8155-acde6c4d46a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.708444 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-dshm" (OuterVolumeSpecName: "dshm") pod "406dcdd6-0d83-4635-8155-acde6c4d46a0" (UID: "406dcdd6-0d83-4635-8155-acde6c4d46a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.708444 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67753ed0-12fe-4da5-a887-f8c9df884fa2" (UID: "67753ed0-12fe-4da5-a887-f8c9df884fa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.708604 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-model-cache" (OuterVolumeSpecName: "model-cache") pod "67753ed0-12fe-4da5-a887-f8c9df884fa2" (UID: "67753ed0-12fe-4da5-a887-f8c9df884fa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.708776 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b9512f-6a84-439e-8ebd-51d8d18f5e88-kube-api-access-268kf" (OuterVolumeSpecName: "kube-api-access-268kf") pod "86b9512f-6a84-439e-8ebd-51d8d18f5e88" (UID: "86b9512f-6a84-439e-8ebd-51d8d18f5e88"). InnerVolumeSpecName "kube-api-access-268kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:13.708837 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708806    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-dshm" (OuterVolumeSpecName: "dshm") pod "67753ed0-12fe-4da5-a887-f8c9df884fa2" (UID: "67753ed0-12fe-4da5-a887-f8c9df884fa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:13.708968 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.708943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406dcdd6-0d83-4635-8155-acde6c4d46a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "406dcdd6-0d83-4635-8155-acde6c4d46a0" (UID: "406dcdd6-0d83-4635-8155-acde6c4d46a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:13.709555 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.709533    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406dcdd6-0d83-4635-8155-acde6c4d46a0-kube-api-access-zkptf" (OuterVolumeSpecName: "kube-api-access-zkptf") pod "406dcdd6-0d83-4635-8155-acde6c4d46a0" (UID: "406dcdd6-0d83-4635-8155-acde6c4d46a0"). InnerVolumeSpecName "kube-api-access-zkptf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:13.709965 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.709946    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67753ed0-12fe-4da5-a887-f8c9df884fa2-kube-api-access-f7qpm" (OuterVolumeSpecName: "kube-api-access-f7qpm") pod "67753ed0-12fe-4da5-a887-f8c9df884fa2" (UID: "67753ed0-12fe-4da5-a887-f8c9df884fa2"). InnerVolumeSpecName "kube-api-access-f7qpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:13.793311 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.793216    2578 generic.go:356] "Generic (PLEG): container finished" podID="67753ed0-12fe-4da5-a887-f8c9df884fa2" containerID="73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9" exitCode=0
Sep 11 22:18:13.793311 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.793274    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"
Sep 11 22:18:13.793512 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.793306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld" event={"ID":"67753ed0-12fe-4da5-a887-f8c9df884fa2","Type":"ContainerDied","Data":"73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9"}
Sep 11 22:18:13.793512 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.793347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld" event={"ID":"67753ed0-12fe-4da5-a887-f8c9df884fa2","Type":"ContainerDied","Data":"b38d35dddd714e72df5bc08f19a45df9c9c7c2c6f445acc332af765b5a46d536"}
Sep 11 22:18:13.793512 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.793362    2578 scope.go:117] "RemoveContainer" containerID="73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9"
Sep 11 22:18:13.795211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.795186    2578 generic.go:356] "Generic (PLEG): container finished" podID="1587f9a4-b86f-4c57-afbd-7330269c26c3" containerID="e1f9ff0086963b1e4c752fe4e746f0fc886b40eeeaf1875f821924253e20a3c9" exitCode=0
Sep 11 22:18:13.795322 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.795267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc" event={"ID":"1587f9a4-b86f-4c57-afbd-7330269c26c3","Type":"ContainerDied","Data":"e1f9ff0086963b1e4c752fe4e746f0fc886b40eeeaf1875f821924253e20a3c9"}
Sep 11 22:18:13.796889 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.796868    2578 generic.go:356] "Generic (PLEG): container finished" podID="406dcdd6-0d83-4635-8155-acde6c4d46a0" containerID="d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3" exitCode=0
Sep 11 22:18:13.796990 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.796928    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"
Sep 11 22:18:13.796990 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.796951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k" event={"ID":"406dcdd6-0d83-4635-8155-acde6c4d46a0","Type":"ContainerDied","Data":"d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3"}
Sep 11 22:18:13.796990 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.796986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k" event={"ID":"406dcdd6-0d83-4635-8155-acde6c4d46a0","Type":"ContainerDied","Data":"9c6f10a3b874f7b77988af3de92bf4c477eee074109f0aa3296a083ebef87444"}
Sep 11 22:18:13.798515 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.798494    2578 generic.go:356] "Generic (PLEG): container finished" podID="86b9512f-6a84-439e-8ebd-51d8d18f5e88" containerID="a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d" exitCode=0
Sep 11 22:18:13.798623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.798540    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"
Sep 11 22:18:13.798623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.798587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m" event={"ID":"86b9512f-6a84-439e-8ebd-51d8d18f5e88","Type":"ContainerDied","Data":"a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d"}
Sep 11 22:18:13.798863 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.798649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m" event={"ID":"86b9512f-6a84-439e-8ebd-51d8d18f5e88","Type":"ContainerDied","Data":"b67f7e21faec5bfbadb7dac3b852043c579b218751b61bfb95a39b9a037686a5"}
Sep 11 22:18:13.803894 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.803797    2578 scope.go:117] "RemoveContainer" containerID="73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9"
Sep 11 22:18:13.804135 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:18:13.804108    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9\": container with ID starting with 73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9 not found: ID does not exist" containerID="73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9"
Sep 11 22:18:13.804208 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.804147    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9"} err="failed to get container status \"73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9\": rpc error: code = NotFound desc = could not find container \"73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9\": container with ID starting with 73a4fd4a1394d30167ad5cef693259da55130b3247600167948ab7ee1f3ce7e9 not found: ID does not exist"
Sep 11 22:18:13.804208 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.804177    2578 scope.go:117] "RemoveContainer" containerID="d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3"
Sep 11 22:18:13.807904 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807874    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.807904 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807898    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807906    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b9512f-6a84-439e-8ebd-51d8d18f5e88-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807914    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b9512f-6a84-439e-8ebd-51d8d18f5e88-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807923    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-268kf\" (UniqueName: \"kubernetes.io/projected/86b9512f-6a84-439e-8ebd-51d8d18f5e88-kube-api-access-268kf\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807935    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406dcdd6-0d83-4635-8155-acde6c4d46a0-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807945    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67753ed0-12fe-4da5-a887-f8c9df884fa2-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807952    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807959    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406dcdd6-0d83-4635-8155-acde6c4d46a0-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807966    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807974    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkptf\" (UniqueName: \"kubernetes.io/projected/406dcdd6-0d83-4635-8155-acde6c4d46a0-kube-api-access-zkptf\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807983    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/67753ed0-12fe-4da5-a887-f8c9df884fa2-kube-api-access-f7qpm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.808050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.807991    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67753ed0-12fe-4da5-a887-f8c9df884fa2-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:13.814259 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.814241    2578 scope.go:117] "RemoveContainer" containerID="d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3"
Sep 11 22:18:13.814551 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:18:13.814532    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3\": container with ID starting with d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3 not found: ID does not exist" containerID="d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3"
Sep 11 22:18:13.814598 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.814559    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3"} err="failed to get container status \"d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3\": rpc error: code = NotFound desc = could not find container \"d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3\": container with ID starting with d02c4c4b0013c614568dcf3855743867fb88aacc5a3221217e26e42537689fc3 not found: ID does not exist"
Sep 11 22:18:13.814598 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.814576    2578 scope.go:117] "RemoveContainer" containerID="a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d"
Sep 11 22:18:13.829802 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.825788    2578 scope.go:117] "RemoveContainer" containerID="a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d"
Sep 11 22:18:13.829802 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:18:13.826583    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d\": container with ID starting with a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d not found: ID does not exist" containerID="a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d"
Sep 11 22:18:13.829802 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.826629    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d"} err="failed to get container status \"a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d\": rpc error: code = NotFound desc = could not find container \"a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d\": container with ID starting with a27e22f466073ef83d9c702d25b0cffd8e184e46f4723e6767c88f4cabf7f34d not found: ID does not exist"
Sep 11 22:18:13.829802 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.827997    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"]
Sep 11 22:18:13.833383 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.833351    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-2-w4-7e2add-kserve-646bb8c57-hkkld"]
Sep 11 22:18:13.840494 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.840469    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"]
Sep 11 22:18:13.844315 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.844294    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/simulator-w4-7e2add-kserve-75ff79b96b-rvk9k"]
Sep 11 22:18:13.853694 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.853666    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"]
Sep 11 22:18:13.857738 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:13.857713    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7e2add/distinct-w4-7e2add-kserve-c8dd594bb-kg85m"]
Sep 11 22:18:14.858210 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:14.858173    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="406dcdd6-0d83-4635-8155-acde6c4d46a0" path="/var/lib/kubelet/pods/406dcdd6-0d83-4635-8155-acde6c4d46a0/volumes"
Sep 11 22:18:14.858943 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:14.858807    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67753ed0-12fe-4da5-a887-f8c9df884fa2" path="/var/lib/kubelet/pods/67753ed0-12fe-4da5-a887-f8c9df884fa2/volumes"
Sep 11 22:18:14.859297 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:14.859278    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86b9512f-6a84-439e-8ebd-51d8d18f5e88" path="/var/lib/kubelet/pods/86b9512f-6a84-439e-8ebd-51d8d18f5e88/volumes"
Sep 11 22:18:14.934116 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:14.934092    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"
Sep 11 22:18:15.121664 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:15.121546    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1587f9a4-b86f-4c57-afbd-7330269c26c3-maas-api-service-ca\") pod \"1587f9a4-b86f-4c57-afbd-7330269c26c3\" (UID: \"1587f9a4-b86f-4c57-afbd-7330269c26c3\") "
Sep 11 22:18:15.121950 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:15.121927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1587f9a4-b86f-4c57-afbd-7330269c26c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1587f9a4-b86f-4c57-afbd-7330269c26c3" (UID: "1587f9a4-b86f-4c57-afbd-7330269c26c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:15.222346 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:15.222308    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1587f9a4-b86f-4c57-afbd-7330269c26c3-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:15.814526 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:15.814496    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"
Sep 11 22:18:15.814728 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:15.814534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc" event={"ID":"1587f9a4-b86f-4c57-afbd-7330269c26c3","Type":"ContainerDied","Data":"fdea98d0bf4295fc1da8d26d2894b92e836230154bb40eb001f31f7fcdffe3dc"}
Sep 11 22:18:15.814728 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:15.814580    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdea98d0bf4295fc1da8d26d2894b92e836230154bb40eb001f31f7fcdffe3dc"
Sep 11 22:18:17.831161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:17.831131    2578 generic.go:356] "Generic (PLEG): container finished" podID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerID="eff66ff9177e6c97dd3a8eb2fc0fd6f136d00d1d817a40682d2915fae1c098e3" exitCode=0
Sep 11 22:18:17.831615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:17.831212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" event={"ID":"605bd98e-cb66-4cd4-b7b3-56570708bcea","Type":"ContainerDied","Data":"eff66ff9177e6c97dd3a8eb2fc0fd6f136d00d1d817a40682d2915fae1c098e3"}
Sep 11 22:18:17.831615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:17.831277    2578 scope.go:117] "RemoveContainer" containerID="403038852de2811c2d2a20dadcc46bdba4cf386690e38709e76ecef4f0b49bda"
Sep 11 22:18:17.908739 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:17.908715    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:18:18.048920 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.048888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmxdk\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-kube-api-access-kmxdk\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049090 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.048943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-certs\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049090 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.048979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-data\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049090 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-envoy\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049090 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049067    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-token\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605bd98e-cb66-4cd4-b7b3-56570708bcea-istiod-ca-cert\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-socket\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049231    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-credential-socket\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049257    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-podinfo\") pod \"605bd98e-cb66-4cd4-b7b3-56570708bcea\" (UID: \"605bd98e-cb66-4cd4-b7b3-56570708bcea\") "
Sep 11 22:18:18.049513 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049347    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-data" (OuterVolumeSpecName: "istio-data") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:18.049513 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:18.049622 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:18.049721 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049621    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.049721 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049675    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.049835 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:18.049957 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.049928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605bd98e-cb66-4cd4-b7b3-56570708bcea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:18.051452 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.051425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-kube-api-access-kmxdk" (OuterVolumeSpecName: "kube-api-access-kmxdk") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "kube-api-access-kmxdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:18.051452 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.051444    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:18:18.051596 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.051434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:18.051777 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.051761    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-token" (OuterVolumeSpecName: "istio-token") pod "605bd98e-cb66-4cd4-b7b3-56570708bcea" (UID: "605bd98e-cb66-4cd4-b7b3-56570708bcea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:18.150744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.150704    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.150744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.150735    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.150744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.150744    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605bd98e-cb66-4cd4-b7b3-56570708bcea-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.150744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.150752    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.151004 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.150760    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605bd98e-cb66-4cd4-b7b3-56570708bcea-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.151004 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.150770    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605bd98e-cb66-4cd4-b7b3-56570708bcea-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.151004 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.150778    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmxdk\" (UniqueName: \"kubernetes.io/projected/605bd98e-cb66-4cd4-b7b3-56570708bcea-kube-api-access-kmxdk\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:18.838822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.838788    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"
Sep 11 22:18:18.839296 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.838828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw" event={"ID":"605bd98e-cb66-4cd4-b7b3-56570708bcea","Type":"ContainerDied","Data":"ec7a933f0b6a246295e413292eafca5d459acce2d92ea49ec09563fa69d0cbb7"}
Sep 11 22:18:18.839296 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.838888    2578 scope.go:117] "RemoveContainer" containerID="eff66ff9177e6c97dd3a8eb2fc0fd6f136d00d1d817a40682d2915fae1c098e3"
Sep 11 22:18:18.862748 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.862719    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"]
Sep 11 22:18:18.868907 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:18.868879    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7e2add-openshift-default-66c4746c9d-g7mqw"]
Sep 11 22:18:20.856802 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:20.856766    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" path="/var/lib/kubelet/pods/605bd98e-cb66-4cd4-b7b3-56570708bcea/volumes"
Sep 11 22:18:22.213747 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:22.213713    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"]
Sep 11 22:18:22.219901 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:22.219868    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5ef59c-29819415-wxn8d"]
Sep 11 22:18:22.396408 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:22.396364    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"]
Sep 11 22:18:22.399241 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:22.399208    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5ef59c-4ksfc"]
Sep 11 22:18:22.856750 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:22.856712    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1587f9a4-b86f-4c57-afbd-7330269c26c3" path="/var/lib/kubelet/pods/1587f9a4-b86f-4c57-afbd-7330269c26c3/volumes"
Sep 11 22:18:22.857061 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:22.857045    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48bf43b-468d-4895-9e33-0c624c6df8a5" path="/var/lib/kubelet/pods/e48bf43b-468d-4895-9e33-0c624c6df8a5/volumes"
Sep 11 22:18:24.320467 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320431    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-354672"]
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320805    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1587f9a4-b86f-4c57-afbd-7330269c26c3" containerName="revoke-keys"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320817    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1587f9a4-b86f-4c57-afbd-7330269c26c3" containerName="revoke-keys"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320829    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b9512f-6a84-439e-8ebd-51d8d18f5e88" containerName="main"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320834    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b9512f-6a84-439e-8ebd-51d8d18f5e88" containerName="main"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320847    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406dcdd6-0d83-4635-8155-acde6c4d46a0" containerName="main"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320852    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="406dcdd6-0d83-4635-8155-acde6c4d46a0" containerName="main"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320860    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320865    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320874    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67753ed0-12fe-4da5-a887-f8c9df884fa2" containerName="main"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320879    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67753ed0-12fe-4da5-a887-f8c9df884fa2" containerName="main"
Sep 11 22:18:24.320884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320889    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy"
Sep 11 22:18:24.321237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320895    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy"
Sep 11 22:18:24.321237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320957    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy"
Sep 11 22:18:24.321237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320965    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b9512f-6a84-439e-8ebd-51d8d18f5e88" containerName="main"
Sep 11 22:18:24.321237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320972    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67753ed0-12fe-4da5-a887-f8c9df884fa2" containerName="main"
Sep 11 22:18:24.321237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320978    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="605bd98e-cb66-4cd4-b7b3-56570708bcea" containerName="istio-proxy"
Sep 11 22:18:24.321237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320984    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1587f9a4-b86f-4c57-afbd-7330269c26c3" containerName="revoke-keys"
Sep 11 22:18:24.321237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.320989    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="406dcdd6-0d83-4635-8155-acde6c4d46a0" containerName="main"
Sep 11 22:18:24.324259 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.324239    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-354672"
Sep 11 22:18:24.326597 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.326574    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h4c6b\""
Sep 11 22:18:24.330162 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.330137    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-354672"]
Sep 11 22:18:24.405253 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.405208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vmx\" (UniqueName: \"kubernetes.io/projected/d4b23f5a-1765-4233-96cc-090c97967f97-kube-api-access-l9vmx\") pod \"test-curl-7684-354672\" (UID: \"d4b23f5a-1765-4233-96cc-090c97967f97\") " pod="opendatahub/test-curl-7684-354672"
Sep 11 22:18:24.506568 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.506521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vmx\" (UniqueName: \"kubernetes.io/projected/d4b23f5a-1765-4233-96cc-090c97967f97-kube-api-access-l9vmx\") pod \"test-curl-7684-354672\" (UID: \"d4b23f5a-1765-4233-96cc-090c97967f97\") " pod="opendatahub/test-curl-7684-354672"
Sep 11 22:18:24.513420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.513391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vmx\" (UniqueName: \"kubernetes.io/projected/d4b23f5a-1765-4233-96cc-090c97967f97-kube-api-access-l9vmx\") pod \"test-curl-7684-354672\" (UID: \"d4b23f5a-1765-4233-96cc-090c97967f97\") " pod="opendatahub/test-curl-7684-354672"
Sep 11 22:18:24.635173 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.635073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-354672"
Sep 11 22:18:24.906384 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:24.906224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-354672"]
Sep 11 22:18:24.908772 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:18:24.908744    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b23f5a_1765_4233_96cc_090c97967f97.slice/crio-eef96d33eec191ae6a08bb801866d1c3d9098f12763e9a3c9b8ebcc157d477bb WatchSource:0}: Error finding container eef96d33eec191ae6a08bb801866d1c3d9098f12763e9a3c9b8ebcc157d477bb: Status 404 returned error can't find the container with id eef96d33eec191ae6a08bb801866d1c3d9098f12763e9a3c9b8ebcc157d477bb
Sep 11 22:18:25.883629 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:25.883587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-354672" event={"ID":"d4b23f5a-1765-4233-96cc-090c97967f97","Type":"ContainerStarted","Data":"59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf"}
Sep 11 22:18:25.883629 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:25.883653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-354672" event={"ID":"d4b23f5a-1765-4233-96cc-090c97967f97","Type":"ContainerStarted","Data":"eef96d33eec191ae6a08bb801866d1c3d9098f12763e9a3c9b8ebcc157d477bb"}
Sep 11 22:18:25.897288 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:25.897232    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7684-354672" podStartSLOduration=1.89721223 podStartE2EDuration="1.89721223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:18:25.896814369 +0000 UTC m=+1567.581761084" watchObservedRunningTime="2026-09-11 22:18:25.89721223 +0000 UTC m=+1567.582158947"
Sep 11 22:18:26.521947 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:26.521908    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-354672"]
Sep 11 22:18:26.526948 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:26.526914    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-354672"]
Sep 11 22:18:26.889623 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:18:26.889592    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7684-354672" container="test-curl-7684-354672"
Sep 11 22:18:27.895483 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:27.895445    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7684-354672" podUID="d4b23f5a-1765-4233-96cc-090c97967f97" containerName="test-curl-7684-354672" containerID="cri-o://59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf" gracePeriod=2
Sep 11 22:18:29.784115 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.784080    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"]
Sep 11 22:18:29.784503 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.784463    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b23f5a-1765-4233-96cc-090c97967f97" containerName="test-curl-7684-354672"
Sep 11 22:18:29.784503 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.784473    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b23f5a-1765-4233-96cc-090c97967f97" containerName="test-curl-7684-354672"
Sep 11 22:18:29.784575 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.784533    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b23f5a-1765-4233-96cc-090c97967f97" containerName="test-curl-7684-354672"
Sep 11 22:18:29.787680 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.787663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"
Sep 11 22:18:29.789521 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.789490    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:18:29.793683 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.793625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"]
Sep 11 22:18:29.857848 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.857812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afe888a-60af-4644-887a-b6bc23947c52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288\" (UID: \"4afe888a-60af-4644-887a-b6bc23947c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"
Sep 11 22:18:29.959353 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.959108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afe888a-60af-4644-887a-b6bc23947c52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288\" (UID: \"4afe888a-60af-4644-887a-b6bc23947c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"
Sep 11 22:18:29.959799 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:29.959775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afe888a-60af-4644-887a-b6bc23947c52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288\" (UID: \"4afe888a-60af-4644-887a-b6bc23947c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"
Sep 11 22:18:30.067284 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.067257    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-354672"
Sep 11 22:18:30.100220 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.100168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"
Sep 11 22:18:30.160199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.160165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9vmx\" (UniqueName: \"kubernetes.io/projected/d4b23f5a-1765-4233-96cc-090c97967f97-kube-api-access-l9vmx\") pod \"d4b23f5a-1765-4233-96cc-090c97967f97\" (UID: \"d4b23f5a-1765-4233-96cc-090c97967f97\") "
Sep 11 22:18:30.164795 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.164758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b23f5a-1765-4233-96cc-090c97967f97-kube-api-access-l9vmx" (OuterVolumeSpecName: "kube-api-access-l9vmx") pod "d4b23f5a-1765-4233-96cc-090c97967f97" (UID: "d4b23f5a-1765-4233-96cc-090c97967f97"). InnerVolumeSpecName "kube-api-access-l9vmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:30.261502 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.261463    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vmx\" (UniqueName: \"kubernetes.io/projected/d4b23f5a-1765-4233-96cc-090c97967f97-kube-api-access-l9vmx\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:30.298983 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.298903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"]
Sep 11 22:18:30.370440 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.370415    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"]
Sep 11 22:18:30.372198 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:18:30.372173    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afe888a_60af_4644_887a_b6bc23947c52.slice/crio-29dd19afa4daba7a4da3b9e4f67480737941aa1c09059224c4a53eafebfe1e58 WatchSource:0}: Error finding container 29dd19afa4daba7a4da3b9e4f67480737941aa1c09059224c4a53eafebfe1e58: Status 404 returned error can't find the container with id 29dd19afa4daba7a4da3b9e4f67480737941aa1c09059224c4a53eafebfe1e58
Sep 11 22:18:30.857800 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.857760    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b23f5a-1765-4233-96cc-090c97967f97" path="/var/lib/kubelet/pods/d4b23f5a-1765-4233-96cc-090c97967f97/volumes"
Sep 11 22:18:30.915250 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.915221    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4b23f5a-1765-4233-96cc-090c97967f97" containerID="59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf" exitCode=137
Sep 11 22:18:30.915427 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.915281    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-354672"
Sep 11 22:18:30.915427 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.915322    2578 scope.go:117] "RemoveContainer" containerID="59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf"
Sep 11 22:18:30.917358 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.917329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288" event={"ID":"4afe888a-60af-4644-887a-b6bc23947c52","Type":"ContainerStarted","Data":"67b9223c70e79477a7cd3c54d85ee1e4d43240cfef7d0a69a0cf6dc49d403434"}
Sep 11 22:18:30.917470 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.917388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288" event={"ID":"4afe888a-60af-4644-887a-b6bc23947c52","Type":"ContainerStarted","Data":"29dd19afa4daba7a4da3b9e4f67480737941aa1c09059224c4a53eafebfe1e58"}
Sep 11 22:18:30.924571 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.924553    2578 scope.go:117] "RemoveContainer" containerID="59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf"
Sep 11 22:18:30.924873 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:18:30.924852    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf\": container with ID starting with 59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf not found: ID does not exist" containerID="59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf"
Sep 11 22:18:30.924942 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.924885    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf"} err="failed to get container status \"59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf\": rpc error: code = NotFound desc = could not find container \"59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf\": container with ID starting with 59fc10d11d17a6e10ea5968f66c1935db74fa8c1db2e0d85152538fc2b767fdf not found: ID does not exist"
Sep 11 22:18:30.934228 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:30.934182    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288" podStartSLOduration=1.934168219 podStartE2EDuration="1.934168219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:18:30.930948745 +0000 UTC m=+1572.615895459" watchObservedRunningTime="2026-09-11 22:18:30.934168219 +0000 UTC m=+1572.619114933"
Sep 11 22:18:31.925458 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:31.925419    2578 generic.go:356] "Generic (PLEG): container finished" podID="4afe888a-60af-4644-887a-b6bc23947c52" containerID="67b9223c70e79477a7cd3c54d85ee1e4d43240cfef7d0a69a0cf6dc49d403434" exitCode=0
Sep 11 22:18:31.925879 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:31.925465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288" event={"ID":"4afe888a-60af-4644-887a-b6bc23947c52","Type":"ContainerDied","Data":"67b9223c70e79477a7cd3c54d85ee1e4d43240cfef7d0a69a0cf6dc49d403434"}
Sep 11 22:18:33.058171 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:33.058144    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"
Sep 11 22:18:33.188463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:33.188370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afe888a-60af-4644-887a-b6bc23947c52-maas-api-service-ca\") pod \"4afe888a-60af-4644-887a-b6bc23947c52\" (UID: \"4afe888a-60af-4644-887a-b6bc23947c52\") "
Sep 11 22:18:33.188787 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:33.188764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afe888a-60af-4644-887a-b6bc23947c52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4afe888a-60af-4644-887a-b6bc23947c52" (UID: "4afe888a-60af-4644-887a-b6bc23947c52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:33.289131 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:33.289097    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afe888a-60af-4644-887a-b6bc23947c52-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:33.938700 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:33.938667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288" event={"ID":"4afe888a-60af-4644-887a-b6bc23947c52","Type":"ContainerDied","Data":"29dd19afa4daba7a4da3b9e4f67480737941aa1c09059224c4a53eafebfe1e58"}
Sep 11 22:18:33.938890 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:33.938700    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"
Sep 11 22:18:33.938890 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:33.938716    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29dd19afa4daba7a4da3b9e4f67480737941aa1c09059224c4a53eafebfe1e58"
Sep 11 22:18:40.356292 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.356256    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"]
Sep 11 22:18:40.356765 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.356492    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2" podUID="dd3ff945-b7fa-4234-a821-975db5e7ad21" containerName="payload-processing" containerID="cri-o://4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7" gracePeriod=30
Sep 11 22:18:40.531428 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.531393    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"]
Sep 11 22:18:40.533942 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.533909    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-38f772-jr288"]
Sep 11 22:18:40.599624 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.599597    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:18:40.659116 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.659025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3ff945-b7fa-4234-a821-975db5e7ad21-plugins-config-volume\") pod \"dd3ff945-b7fa-4234-a821-975db5e7ad21\" (UID: \"dd3ff945-b7fa-4234-a821-975db5e7ad21\") "
Sep 11 22:18:40.659288 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.659116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl98m\" (UniqueName: \"kubernetes.io/projected/dd3ff945-b7fa-4234-a821-975db5e7ad21-kube-api-access-gl98m\") pod \"dd3ff945-b7fa-4234-a821-975db5e7ad21\" (UID: \"dd3ff945-b7fa-4234-a821-975db5e7ad21\") "
Sep 11 22:18:40.659480 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.659454    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3ff945-b7fa-4234-a821-975db5e7ad21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd3ff945-b7fa-4234-a821-975db5e7ad21" (UID: "dd3ff945-b7fa-4234-a821-975db5e7ad21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:40.661370 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.661347    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3ff945-b7fa-4234-a821-975db5e7ad21-kube-api-access-gl98m" (OuterVolumeSpecName: "kube-api-access-gl98m") pod "dd3ff945-b7fa-4234-a821-975db5e7ad21" (UID: "dd3ff945-b7fa-4234-a821-975db5e7ad21"). InnerVolumeSpecName "kube-api-access-gl98m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:40.760498 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.760461    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl98m\" (UniqueName: \"kubernetes.io/projected/dd3ff945-b7fa-4234-a821-975db5e7ad21-kube-api-access-gl98m\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:40.760498 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.760492    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3ff945-b7fa-4234-a821-975db5e7ad21-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:40.857765 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.857730    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4afe888a-60af-4644-887a-b6bc23947c52" path="/var/lib/kubelet/pods/4afe888a-60af-4644-887a-b6bc23947c52/volumes"
Sep 11 22:18:40.978970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.978931    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd3ff945-b7fa-4234-a821-975db5e7ad21" containerID="4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7" exitCode=0
Sep 11 22:18:40.979145 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.978984    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"
Sep 11 22:18:40.979145 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.979005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2" event={"ID":"dd3ff945-b7fa-4234-a821-975db5e7ad21","Type":"ContainerDied","Data":"4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7"}
Sep 11 22:18:40.979145 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.979064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2" event={"ID":"dd3ff945-b7fa-4234-a821-975db5e7ad21","Type":"ContainerDied","Data":"0e915fd2b7e1edcc23e24e0dcab6bcb8fe689a00eed6c54a23d28fee632a85ac"}
Sep 11 22:18:40.979145 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.979091    2578 scope.go:117] "RemoveContainer" containerID="4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7"
Sep 11 22:18:40.989201 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.989179    2578 scope.go:117] "RemoveContainer" containerID="4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7"
Sep 11 22:18:40.989537 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:18:40.989514    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7\": container with ID starting with 4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7 not found: ID does not exist" containerID="4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7"
Sep 11 22:18:40.989593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.989547    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7"} err="failed to get container status \"4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7\": rpc error: code = NotFound desc = could not find container \"4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7\": container with ID starting with 4efd961372293a11788e4be14fc4465c171b99662f771d6aa6690877cb1fd8a7 not found: ID does not exist"
Sep 11 22:18:40.997260 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:40.997233    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"]
Sep 11 22:18:41.003689 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:41.003662    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-38f772-6dfb667fc7-bvzw2"]
Sep 11 22:18:42.862439 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:42.862391    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd3ff945-b7fa-4234-a821-975db5e7ad21" path="/var/lib/kubelet/pods/dd3ff945-b7fa-4234-a821-975db5e7ad21/volumes"
Sep 11 22:18:45.318050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.318006    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj" podUID="b967773c-936d-4e3a-82cc-5c9a7040dc67" containerName="main" containerID="cri-o://2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f" gracePeriod=45
Sep 11 22:18:45.568849 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.568788    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:18:45.708907 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.708871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-model-cache\") pod \"b967773c-936d-4e3a-82cc-5c9a7040dc67\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") "
Sep 11 22:18:45.709081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.708955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-tmp-dir\") pod \"b967773c-936d-4e3a-82cc-5c9a7040dc67\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") "
Sep 11 22:18:45.709081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.708982    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b967773c-936d-4e3a-82cc-5c9a7040dc67-tls-certs\") pod \"b967773c-936d-4e3a-82cc-5c9a7040dc67\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") "
Sep 11 22:18:45.709081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709027    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lt75\" (UniqueName: \"kubernetes.io/projected/b967773c-936d-4e3a-82cc-5c9a7040dc67-kube-api-access-7lt75\") pod \"b967773c-936d-4e3a-82cc-5c9a7040dc67\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") "
Sep 11 22:18:45.709081 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-home\") pod \"b967773c-936d-4e3a-82cc-5c9a7040dc67\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") "
Sep 11 22:18:45.709345 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-dshm\") pod \"b967773c-936d-4e3a-82cc-5c9a7040dc67\" (UID: \"b967773c-936d-4e3a-82cc-5c9a7040dc67\") "
Sep 11 22:18:45.709345 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709178    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-model-cache" (OuterVolumeSpecName: "model-cache") pod "b967773c-936d-4e3a-82cc-5c9a7040dc67" (UID: "b967773c-936d-4e3a-82cc-5c9a7040dc67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:45.709345 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b967773c-936d-4e3a-82cc-5c9a7040dc67" (UID: "b967773c-936d-4e3a-82cc-5c9a7040dc67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:45.709492 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-home" (OuterVolumeSpecName: "home") pod "b967773c-936d-4e3a-82cc-5c9a7040dc67" (UID: "b967773c-936d-4e3a-82cc-5c9a7040dc67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:45.709492 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709406    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:45.709492 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709421    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:45.709492 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.709433    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:45.711325 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.711301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b967773c-936d-4e3a-82cc-5c9a7040dc67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b967773c-936d-4e3a-82cc-5c9a7040dc67" (UID: "b967773c-936d-4e3a-82cc-5c9a7040dc67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:45.711420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.711353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-dshm" (OuterVolumeSpecName: "dshm") pod "b967773c-936d-4e3a-82cc-5c9a7040dc67" (UID: "b967773c-936d-4e3a-82cc-5c9a7040dc67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:45.711458 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.711421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b967773c-936d-4e3a-82cc-5c9a7040dc67-kube-api-access-7lt75" (OuterVolumeSpecName: "kube-api-access-7lt75") pod "b967773c-936d-4e3a-82cc-5c9a7040dc67" (UID: "b967773c-936d-4e3a-82cc-5c9a7040dc67"). InnerVolumeSpecName "kube-api-access-7lt75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:45.810139 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.810100    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b967773c-936d-4e3a-82cc-5c9a7040dc67-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:45.810139 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.810134    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lt75\" (UniqueName: \"kubernetes.io/projected/b967773c-936d-4e3a-82cc-5c9a7040dc67-kube-api-access-7lt75\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:45.810139 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:45.810146    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b967773c-936d-4e3a-82cc-5c9a7040dc67-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:18:46.010394 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.010351    2578 generic.go:356] "Generic (PLEG): container finished" podID="b967773c-936d-4e3a-82cc-5c9a7040dc67" containerID="2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f" exitCode=0
Sep 11 22:18:46.010575 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.010402    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"
Sep 11 22:18:46.010575 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.010436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj" event={"ID":"b967773c-936d-4e3a-82cc-5c9a7040dc67","Type":"ContainerDied","Data":"2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f"}
Sep 11 22:18:46.010575 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.010481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj" event={"ID":"b967773c-936d-4e3a-82cc-5c9a7040dc67","Type":"ContainerDied","Data":"5438df968dc0bcf3ee5e25e79da9d27b972d91033acb69e230e2d4fb8aebaf9c"}
Sep 11 22:18:46.010575 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.010499    2578 scope.go:117] "RemoveContainer" containerID="2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f"
Sep 11 22:18:46.021034 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.021014    2578 scope.go:117] "RemoveContainer" containerID="2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f"
Sep 11 22:18:46.021295 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:18:46.021276    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f\": container with ID starting with 2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f not found: ID does not exist" containerID="2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f"
Sep 11 22:18:46.021339 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.021304    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f"} err="failed to get container status \"2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f\": rpc error: code = NotFound desc = could not find container \"2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f\": container with ID starting with 2e836502a037c1846ef2c7300e10728be4caf377980990c0180b56280c08b99f not found: ID does not exist"
Sep 11 22:18:46.033028 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.033003    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"]
Sep 11 22:18:46.036913 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.036891    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/auth-test-model-966527-kserve-78764fdf8c-vvwhj"]
Sep 11 22:18:46.856927 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:18:46.856892    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b967773c-936d-4e3a-82cc-5c9a7040dc67" path="/var/lib/kubelet/pods/b967773c-936d-4e3a-82cc-5c9a7040dc67/volumes"
Sep 11 22:19:16.580502 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.580404    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"]
Sep 11 22:19:16.581043 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.580993    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afe888a-60af-4644-887a-b6bc23947c52" containerName="revoke-keys"
Sep 11 22:19:16.581043 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.581013    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afe888a-60af-4644-887a-b6bc23947c52" containerName="revoke-keys"
Sep 11 22:19:16.581043 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.581038    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd3ff945-b7fa-4234-a821-975db5e7ad21" containerName="payload-processing"
Sep 11 22:19:16.581211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.581047    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd3ff945-b7fa-4234-a821-975db5e7ad21" containerName="payload-processing"
Sep 11 22:19:16.581211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.581060    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b967773c-936d-4e3a-82cc-5c9a7040dc67" containerName="main"
Sep 11 22:19:16.581211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.581067    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b967773c-936d-4e3a-82cc-5c9a7040dc67" containerName="main"
Sep 11 22:19:16.581211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.581133    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd3ff945-b7fa-4234-a821-975db5e7ad21" containerName="payload-processing"
Sep 11 22:19:16.581211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.581146    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afe888a-60af-4644-887a-b6bc23947c52" containerName="revoke-keys"
Sep 11 22:19:16.581211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.581153    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b967773c-936d-4e3a-82cc-5c9a7040dc67" containerName="main"
Sep 11 22:19:16.584227 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.584206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.586975 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.586937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"default-dockercfg-tc5dd\""
Sep 11 22:19:16.587112 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.586982    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"openshift-service-ca.crt\""
Sep 11 22:19:16.587112 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.587008    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"kube-root-ca.crt\""
Sep 11 22:19:16.587112 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.587039    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-966527\"/\"sub-test-model-966527-kserve-self-signed-certs\""
Sep 11 22:19:16.592058 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.592036    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"]
Sep 11 22:19:16.682266 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.682231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-home\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.682466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.682298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a6841b-963f-42d4-9e23-17fbb221036e-tls-certs\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.682466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.682332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-tmp-dir\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.682466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.682411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-model-cache\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.682466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.682458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8twz\" (UniqueName: \"kubernetes.io/projected/44a6841b-963f-42d4-9e23-17fbb221036e-kube-api-access-g8twz\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.682665 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.682510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-dshm\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.783579 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.783538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a6841b-963f-42d4-9e23-17fbb221036e-tls-certs\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.783816 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.783586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-tmp-dir\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.783816 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.783628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-model-cache\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.783816 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.783690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8twz\" (UniqueName: \"kubernetes.io/projected/44a6841b-963f-42d4-9e23-17fbb221036e-kube-api-access-g8twz\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.783816 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.783749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-dshm\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.784050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.783886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-home\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.784113 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.784061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-tmp-dir\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.784160 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.784140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-model-cache\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.784249 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.784225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-home\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.785936 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.785915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-dshm\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.786177 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.786159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a6841b-963f-42d4-9e23-17fbb221036e-tls-certs\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.791313 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.791293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8twz\" (UniqueName: \"kubernetes.io/projected/44a6841b-963f-42d4-9e23-17fbb221036e-kube-api-access-g8twz\") pod \"sub-test-model-966527-kserve-b59645965-vqh68\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") " pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:16.894941 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:16.894842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:17.166227 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:17.166200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"]
Sep 11 22:19:17.168595 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:19:17.168566    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44a6841b_963f_42d4_9e23_17fbb221036e.slice/crio-0f2a5a3e377c5893f916c64af284dc0a226e0b02039529de38bec244b9fc68d0 WatchSource:0}: Error finding container 0f2a5a3e377c5893f916c64af284dc0a226e0b02039529de38bec244b9fc68d0: Status 404 returned error can't find the container with id 0f2a5a3e377c5893f916c64af284dc0a226e0b02039529de38bec244b9fc68d0
Sep 11 22:19:17.212086 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:17.212042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68" event={"ID":"44a6841b-963f-42d4-9e23-17fbb221036e","Type":"ContainerStarted","Data":"0f2a5a3e377c5893f916c64af284dc0a226e0b02039529de38bec244b9fc68d0"}
Sep 11 22:19:18.219525 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:18.219491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68" event={"ID":"44a6841b-963f-42d4-9e23-17fbb221036e","Type":"ContainerStarted","Data":"35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1"}
Sep 11 22:19:18.236378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:18.236328    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68" podStartSLOduration=2.23631214 podStartE2EDuration="2.23631214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:19:18.234234582 +0000 UTC m=+1619.919181295" watchObservedRunningTime="2026-09-11 22:19:18.23631214 +0000 UTC m=+1619.921258854"
Sep 11 22:19:26.895153 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:26.895116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:26.895585 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:26.895164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:26.907885 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:26.907864    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:19:27.283154 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:19:27.283120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:20:00.163186 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:00.163145    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"]
Sep 11 22:20:08.385624 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.385578    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"]
Sep 11 22:20:08.389105 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.389084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.391303 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.391278    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"openshift-service-ca.crt\""
Sep 11 22:20:08.391413 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.391389    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"kube-root-ca.crt\""
Sep 11 22:20:08.391913 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.391895    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"rate-test-model-d09c96-kserve-self-signed-certs\""
Sep 11 22:20:08.392011 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.391955    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"default-dockercfg-jp7v5\""
Sep 11 22:20:08.399712 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.399688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"]
Sep 11 22:20:08.457513 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.457471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tmp-dir\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.457914 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.457517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-home\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.457914 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.457610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8tt\" (UniqueName: \"kubernetes.io/projected/a5e6eed0-7ead-4050-b2fb-2735a75098ad-kube-api-access-cx8tt\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.457914 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.457655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-model-cache\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.457914 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.457686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tls-certs\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.457914 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.457744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-dshm\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.558711 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.558672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tmp-dir\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.558711 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.558726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-home\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.558984 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.558787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8tt\" (UniqueName: \"kubernetes.io/projected/a5e6eed0-7ead-4050-b2fb-2735a75098ad-kube-api-access-cx8tt\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.558984 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.558813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-model-cache\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.558984 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.558841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tls-certs\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.558984 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.558888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-dshm\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.559204 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.559091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tmp-dir\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.559204 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.559159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-home\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.559204 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.559192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-model-cache\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.561071 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.561047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-dshm\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.561366 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.561346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tls-certs\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.566123 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.566102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8tt\" (UniqueName: \"kubernetes.io/projected/a5e6eed0-7ead-4050-b2fb-2735a75098ad-kube-api-access-cx8tt\") pod \"rate-test-model-d09c96-kserve-668667765f-dtcvn\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.700316 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.700272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:08.970924 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:08.970890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"]
Sep 11 22:20:08.972488 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:20:08.972460    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5e6eed0_7ead_4050_b2fb_2735a75098ad.slice/crio-eaef4918455b74967cd43d2e99788321700c68d96c72ab5f7afd02bf1bca10d6 WatchSource:0}: Error finding container eaef4918455b74967cd43d2e99788321700c68d96c72ab5f7afd02bf1bca10d6: Status 404 returned error can't find the container with id eaef4918455b74967cd43d2e99788321700c68d96c72ab5f7afd02bf1bca10d6
Sep 11 22:20:09.518668 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:09.518614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn" event={"ID":"a5e6eed0-7ead-4050-b2fb-2735a75098ad","Type":"ContainerStarted","Data":"77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320"}
Sep 11 22:20:09.519098 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:09.518677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn" event={"ID":"a5e6eed0-7ead-4050-b2fb-2735a75098ad","Type":"ContainerStarted","Data":"eaef4918455b74967cd43d2e99788321700c68d96c72ab5f7afd02bf1bca10d6"}
Sep 11 22:20:09.536951 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:09.536898    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn" podStartSLOduration=1.5368827189999998 podStartE2EDuration="1.536882719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:20:09.533752042 +0000 UTC m=+1671.218698756" watchObservedRunningTime="2026-09-11 22:20:09.536882719 +0000 UTC m=+1671.221829513"
Sep 11 22:20:15.183464 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.183417    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68" podUID="44a6841b-963f-42d4-9e23-17fbb221036e" containerName="main" containerID="cri-o://35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1" gracePeriod=45
Sep 11 22:20:15.433804 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.433743    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:20:15.528454 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528421    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-dshm\") pod \"44a6841b-963f-42d4-9e23-17fbb221036e\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") "
Sep 11 22:20:15.528661 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-tmp-dir\") pod \"44a6841b-963f-42d4-9e23-17fbb221036e\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") "
Sep 11 22:20:15.528661 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528517    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8twz\" (UniqueName: \"kubernetes.io/projected/44a6841b-963f-42d4-9e23-17fbb221036e-kube-api-access-g8twz\") pod \"44a6841b-963f-42d4-9e23-17fbb221036e\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") "
Sep 11 22:20:15.528661 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-model-cache\") pod \"44a6841b-963f-42d4-9e23-17fbb221036e\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") "
Sep 11 22:20:15.528661 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-home\") pod \"44a6841b-963f-42d4-9e23-17fbb221036e\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") "
Sep 11 22:20:15.528661 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a6841b-963f-42d4-9e23-17fbb221036e-tls-certs\") pod \"44a6841b-963f-42d4-9e23-17fbb221036e\" (UID: \"44a6841b-963f-42d4-9e23-17fbb221036e\") "
Sep 11 22:20:15.528955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44a6841b-963f-42d4-9e23-17fbb221036e" (UID: "44a6841b-963f-42d4-9e23-17fbb221036e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:20:15.528955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528848    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-model-cache" (OuterVolumeSpecName: "model-cache") pod "44a6841b-963f-42d4-9e23-17fbb221036e" (UID: "44a6841b-963f-42d4-9e23-17fbb221036e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:20:15.528955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.528931    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-home" (OuterVolumeSpecName: "home") pod "44a6841b-963f-42d4-9e23-17fbb221036e" (UID: "44a6841b-963f-42d4-9e23-17fbb221036e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:20:15.529083 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.529059    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:20:15.529083 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.529078    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:20:15.529185 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.529093    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:20:15.530768 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.530742    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a6841b-963f-42d4-9e23-17fbb221036e-kube-api-access-g8twz" (OuterVolumeSpecName: "kube-api-access-g8twz") pod "44a6841b-963f-42d4-9e23-17fbb221036e" (UID: "44a6841b-963f-42d4-9e23-17fbb221036e"). InnerVolumeSpecName "kube-api-access-g8twz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:20:15.530885 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.530774    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-dshm" (OuterVolumeSpecName: "dshm") pod "44a6841b-963f-42d4-9e23-17fbb221036e" (UID: "44a6841b-963f-42d4-9e23-17fbb221036e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:20:15.530885 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.530773    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a6841b-963f-42d4-9e23-17fbb221036e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44a6841b-963f-42d4-9e23-17fbb221036e" (UID: "44a6841b-963f-42d4-9e23-17fbb221036e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:20:15.556956 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.556925    2578 generic.go:356] "Generic (PLEG): container finished" podID="44a6841b-963f-42d4-9e23-17fbb221036e" containerID="35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1" exitCode=0
Sep 11 22:20:15.557087 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.556965    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"
Sep 11 22:20:15.557087 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.557010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68" event={"ID":"44a6841b-963f-42d4-9e23-17fbb221036e","Type":"ContainerDied","Data":"35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1"}
Sep 11 22:20:15.557087 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.557049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68" event={"ID":"44a6841b-963f-42d4-9e23-17fbb221036e","Type":"ContainerDied","Data":"0f2a5a3e377c5893f916c64af284dc0a226e0b02039529de38bec244b9fc68d0"}
Sep 11 22:20:15.557087 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.557067    2578 scope.go:117] "RemoveContainer" containerID="35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1"
Sep 11 22:20:15.567491 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.567476    2578 scope.go:117] "RemoveContainer" containerID="35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1"
Sep 11 22:20:15.567785 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:20:15.567766    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1\": container with ID starting with 35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1 not found: ID does not exist" containerID="35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1"
Sep 11 22:20:15.567833 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.567795    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1"} err="failed to get container status \"35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1\": rpc error: code = NotFound desc = could not find container \"35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1\": container with ID starting with 35a3eee9d0efd85ecd6f128672ad0e8121dcc79e2cd9b96019b2aa07f2496de1 not found: ID does not exist"
Sep 11 22:20:15.578348 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.578327    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"]
Sep 11 22:20:15.586614 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.586593    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-966527/sub-test-model-966527-kserve-b59645965-vqh68"]
Sep 11 22:20:15.630133 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.630108    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8twz\" (UniqueName: \"kubernetes.io/projected/44a6841b-963f-42d4-9e23-17fbb221036e-kube-api-access-g8twz\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:20:15.630133 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.630130    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a6841b-963f-42d4-9e23-17fbb221036e-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:20:15.630263 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:15.630142    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a6841b-963f-42d4-9e23-17fbb221036e-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:20:16.856570 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:16.856533    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a6841b-963f-42d4-9e23-17fbb221036e" path="/var/lib/kubelet/pods/44a6841b-963f-42d4-9e23-17fbb221036e/volumes"
Sep 11 22:20:18.701020 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:18.700982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:18.701415 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:18.701043    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:18.713728 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:18.713697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:20:19.178605 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:19.178573    2578 scope.go:117] "RemoveContainer" containerID="cf7448b8107d6ef74348eb0dd02cc1c76aa23d96b88f14172297f437e3e6d9f4"
Sep 11 22:20:19.592122 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:20:19.592041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:21:19.206806 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:19.206774    2578 scope.go:117] "RemoveContainer" containerID="d9f0d48a27ad6275f2102c9d9a0b63c5a56b35649c946982a4e675504ef88806"
Sep 11 22:21:19.215308 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:19.215286    2578 scope.go:117] "RemoveContainer" containerID="56f4bfa2b2b43b0f493908dad4935c34028ff7c4dfeb7eb6aa2753ea2c508c55"
Sep 11 22:21:19.902088 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:19.902048    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"]
Sep 11 22:21:34.922023 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:34.921981    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn" podUID="a5e6eed0-7ead-4050-b2fb-2735a75098ad" containerName="main" containerID="cri-o://77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320" gracePeriod=45
Sep 11 22:21:35.171201 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.171178    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:21:35.261798 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.261717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tls-certs\") pod \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") "
Sep 11 22:21:35.261798 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.261774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-home\") pod \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") "
Sep 11 22:21:35.262013 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.261802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-dshm\") pod \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") "
Sep 11 22:21:35.262013 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.261902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx8tt\" (UniqueName: \"kubernetes.io/projected/a5e6eed0-7ead-4050-b2fb-2735a75098ad-kube-api-access-cx8tt\") pod \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") "
Sep 11 22:21:35.262013 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.261952    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tmp-dir\") pod \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") "
Sep 11 22:21:35.262178 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.262015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-model-cache\") pod \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\" (UID: \"a5e6eed0-7ead-4050-b2fb-2735a75098ad\") "
Sep 11 22:21:35.262293 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.262259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5e6eed0-7ead-4050-b2fb-2735a75098ad" (UID: "a5e6eed0-7ead-4050-b2fb-2735a75098ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:21:35.262358 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.262331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5e6eed0-7ead-4050-b2fb-2735a75098ad" (UID: "a5e6eed0-7ead-4050-b2fb-2735a75098ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:21:35.262418 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.262361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-home" (OuterVolumeSpecName: "home") pod "a5e6eed0-7ead-4050-b2fb-2735a75098ad" (UID: "a5e6eed0-7ead-4050-b2fb-2735a75098ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:21:35.264132 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.264100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e6eed0-7ead-4050-b2fb-2735a75098ad-kube-api-access-cx8tt" (OuterVolumeSpecName: "kube-api-access-cx8tt") pod "a5e6eed0-7ead-4050-b2fb-2735a75098ad" (UID: "a5e6eed0-7ead-4050-b2fb-2735a75098ad"). InnerVolumeSpecName "kube-api-access-cx8tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:21:35.264240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.264132    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-dshm" (OuterVolumeSpecName: "dshm") pod "a5e6eed0-7ead-4050-b2fb-2735a75098ad" (UID: "a5e6eed0-7ead-4050-b2fb-2735a75098ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:21:35.264240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.264179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5e6eed0-7ead-4050-b2fb-2735a75098ad" (UID: "a5e6eed0-7ead-4050-b2fb-2735a75098ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:21:35.363039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.363004    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:21:35.363039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.363029    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:21:35.363039 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.363042    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx8tt\" (UniqueName: \"kubernetes.io/projected/a5e6eed0-7ead-4050-b2fb-2735a75098ad-kube-api-access-cx8tt\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:21:35.363280 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.363076    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:21:35.363280 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.363091    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e6eed0-7ead-4050-b2fb-2735a75098ad-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:21:35.363280 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:35.363101    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e6eed0-7ead-4050-b2fb-2735a75098ad-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:21:36.016283 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.016246    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5e6eed0-7ead-4050-b2fb-2735a75098ad" containerID="77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320" exitCode=0
Sep 11 22:21:36.016729 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.016294    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"
Sep 11 22:21:36.016729 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.016326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn" event={"ID":"a5e6eed0-7ead-4050-b2fb-2735a75098ad","Type":"ContainerDied","Data":"77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320"}
Sep 11 22:21:36.016729 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.016373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn" event={"ID":"a5e6eed0-7ead-4050-b2fb-2735a75098ad","Type":"ContainerDied","Data":"eaef4918455b74967cd43d2e99788321700c68d96c72ab5f7afd02bf1bca10d6"}
Sep 11 22:21:36.016729 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.016390    2578 scope.go:117] "RemoveContainer" containerID="77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320"
Sep 11 22:21:36.025730 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.025711    2578 scope.go:117] "RemoveContainer" containerID="77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320"
Sep 11 22:21:36.026016 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:21:36.025997    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320\": container with ID starting with 77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320 not found: ID does not exist" containerID="77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320"
Sep 11 22:21:36.026063 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.026027    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320"} err="failed to get container status \"77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320\": rpc error: code = NotFound desc = could not find container \"77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320\": container with ID starting with 77dce82253d0a2daee37bf9986a21ee760828389664135a077aaf26847a6d320 not found: ID does not exist"
Sep 11 22:21:36.038371 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.038341    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"]
Sep 11 22:21:36.047607 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.047582    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/rate-test-model-d09c96-kserve-668667765f-dtcvn"]
Sep 11 22:21:36.857022 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:36.856987    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e6eed0-7ead-4050-b2fb-2735a75098ad" path="/var/lib/kubelet/pods/a5e6eed0-7ead-4050-b2fb-2735a75098ad/volumes"
Sep 11 22:21:56.904623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.904586    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"]
Sep 11 22:21:56.905094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.905019    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a6841b-963f-42d4-9e23-17fbb221036e" containerName="main"
Sep 11 22:21:56.905094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.905036    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a6841b-963f-42d4-9e23-17fbb221036e" containerName="main"
Sep 11 22:21:56.905094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.905054    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e6eed0-7ead-4050-b2fb-2735a75098ad" containerName="main"
Sep 11 22:21:56.905094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.905060    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e6eed0-7ead-4050-b2fb-2735a75098ad" containerName="main"
Sep 11 22:21:56.905226 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.905129    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e6eed0-7ead-4050-b2fb-2735a75098ad" containerName="main"
Sep 11 22:21:56.905226 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.905137    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a6841b-963f-42d4-9e23-17fbb221036e" containerName="main"
Sep 11 22:21:56.908196 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.908179    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:56.910006 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.909990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dbaf52\""
Sep 11 22:21:56.920466 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:56.920442    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"]
Sep 11 22:21:57.060543 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:57.060442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2ndk\" (UniqueName: \"kubernetes.io/projected/55a7eaed-ca97-48b5-8878-a33984e8e6e8-kube-api-access-r2ndk\") pod \"payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm\" (UID: \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:57.060543 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:57.060528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a7eaed-ca97-48b5-8878-a33984e8e6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm\" (UID: \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:57.161644 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:57.161599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2ndk\" (UniqueName: \"kubernetes.io/projected/55a7eaed-ca97-48b5-8878-a33984e8e6e8-kube-api-access-r2ndk\") pod \"payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm\" (UID: \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:57.161815 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:57.161722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a7eaed-ca97-48b5-8878-a33984e8e6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm\" (UID: \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:57.162440 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:57.162420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a7eaed-ca97-48b5-8878-a33984e8e6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm\" (UID: \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:57.169114 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:57.169094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2ndk\" (UniqueName: \"kubernetes.io/projected/55a7eaed-ca97-48b5-8878-a33984e8e6e8-kube-api-access-r2ndk\") pod \"payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm\" (UID: \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:57.219284 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:57.219254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:57.495857 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:57.495822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"]
Sep 11 22:21:57.499355 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:21:57.499322    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a7eaed_ca97_48b5_8878_a33984e8e6e8.slice/crio-3e72315f10cea9645b9b696d6d727c715f0810ad89ed38685016b386e04d05c1 WatchSource:0}: Error finding container 3e72315f10cea9645b9b696d6d727c715f0810ad89ed38685016b386e04d05c1: Status 404 returned error can't find the container with id 3e72315f10cea9645b9b696d6d727c715f0810ad89ed38685016b386e04d05c1
Sep 11 22:21:58.142896 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:58.142857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm" event={"ID":"55a7eaed-ca97-48b5-8878-a33984e8e6e8","Type":"ContainerStarted","Data":"2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a"}
Sep 11 22:21:58.143380 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:58.142912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm" event={"ID":"55a7eaed-ca97-48b5-8878-a33984e8e6e8","Type":"ContainerStarted","Data":"3e72315f10cea9645b9b696d6d727c715f0810ad89ed38685016b386e04d05c1"}
Sep 11 22:21:58.143380 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:58.142983    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:21:58.163410 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:21:58.163345    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm" podStartSLOduration=2.163328845 podStartE2EDuration="2.163328845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:21:58.161196109 +0000 UTC m=+1779.846142833" watchObservedRunningTime="2026-09-11 22:21:58.163328845 +0000 UTC m=+1779.848275555"
Sep 11 22:22:09.150930 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:09.150899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:22:27.679429 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.679392    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"]
Sep 11 22:22:27.683420 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.683402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.686397 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.686372    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6fe4c2\"/\"kube-root-ca.crt\""
Sep 11 22:22:27.686615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.686408    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6fe4c2\"/\"openshift-service-ca.crt\""
Sep 11 22:22:27.686615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.686382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6fe4c2\"/\"default-dockercfg-tm2v5\""
Sep 11 22:22:27.686615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.686374    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6fe4c2\"/\"ipp-route-6fe4c2-kserve-self-signed-certs\""
Sep 11 22:22:27.692251 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.692226    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"]
Sep 11 22:22:27.734730 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.734694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tmp-dir\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.734912 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.734736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkph\" (UniqueName: \"kubernetes.io/projected/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-kube-api-access-jmkph\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.734912 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.734830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-home\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.734912 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.734862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-dshm\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.735078 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.734943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-model-cache\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.735078 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.735018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tls-certs\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.835978 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.835936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-model-cache\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.836176 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.835998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tls-certs\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.836176 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.836024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tmp-dir\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.836176 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.836042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkph\" (UniqueName: \"kubernetes.io/projected/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-kube-api-access-jmkph\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.836176 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.836092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-home\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.836176 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.836119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-dshm\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.836473 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.836366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-model-cache\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.836473 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.836459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-home\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.836557 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.836503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tmp-dir\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.838361 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.838337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-dshm\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.838646 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.838617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tls-certs\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.843385 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.843365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkph\" (UniqueName: \"kubernetes.io/projected/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-kube-api-access-jmkph\") pod \"ipp-route-6fe4c2-kserve-84b5d69579-cz2h5\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") " pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:27.994159 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:27.994065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:28.268667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:28.268622    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"]
Sep 11 22:22:28.269672 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:22:28.269617    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod899ecd8c_27ca_4af6_8fb4_ac66419f95c0.slice/crio-8c6d76cfd8b22e2291b47624e13fb2906dc2cb5e87f277d0de335564790a88a8 WatchSource:0}: Error finding container 8c6d76cfd8b22e2291b47624e13fb2906dc2cb5e87f277d0de335564790a88a8: Status 404 returned error can't find the container with id 8c6d76cfd8b22e2291b47624e13fb2906dc2cb5e87f277d0de335564790a88a8
Sep 11 22:22:28.315143 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:28.315113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5" event={"ID":"899ecd8c-27ca-4af6-8fb4-ac66419f95c0","Type":"ContainerStarted","Data":"8c6d76cfd8b22e2291b47624e13fb2906dc2cb5e87f277d0de335564790a88a8"}
Sep 11 22:22:29.322124 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:29.322089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5" event={"ID":"899ecd8c-27ca-4af6-8fb4-ac66419f95c0","Type":"ContainerStarted","Data":"9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568"}
Sep 11 22:22:37.994891 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:37.994849    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:37.995298 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:37.994938    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:38.007510 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:38.007477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:22:38.031035 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:38.030982    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5" podStartSLOduration=11.030965828 podStartE2EDuration="11.030965828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:22:29.35763012 +0000 UTC m=+1811.042576847" watchObservedRunningTime="2026-09-11 22:22:38.030965828 +0000 UTC m=+1819.715912541"
Sep 11 22:22:38.386103 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:22:38.386027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:23:08.483397 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.483351    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"]
Sep 11 22:23:08.487436 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.487408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:08.489449 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.489423    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1688a8-dockercfg-9rj7c\""
Sep 11 22:23:08.489575 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.489472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1688a8\""
Sep 11 22:23:08.494569 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.494525    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"]
Sep 11 22:23:08.498615 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.498590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"]
Sep 11 22:23:08.498765 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.498696    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:08.504013 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.503979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8r5w\" (UniqueName: \"kubernetes.io/projected/7a33ad9e-488b-4002-88b0-18af09ad5a82-kube-api-access-c8r5w\") pod \"payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx\" (UID: \"7a33ad9e-488b-4002-88b0-18af09ad5a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:08.504198 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.504057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a33ad9e-488b-4002-88b0-18af09ad5a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx\" (UID: \"7a33ad9e-488b-4002-88b0-18af09ad5a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:08.510459 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.510416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"]
Sep 11 22:23:08.604820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.604784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae815d4d-cf31-4a5a-9705-a218990a4bb5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw\" (UID: \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:08.605007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.604844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8r5w\" (UniqueName: \"kubernetes.io/projected/7a33ad9e-488b-4002-88b0-18af09ad5a82-kube-api-access-c8r5w\") pod \"payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx\" (UID: \"7a33ad9e-488b-4002-88b0-18af09ad5a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:08.605007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.604906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nvc\" (UniqueName: \"kubernetes.io/projected/ae815d4d-cf31-4a5a-9705-a218990a4bb5-kube-api-access-r9nvc\") pod \"payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw\" (UID: \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:08.605007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.604985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a33ad9e-488b-4002-88b0-18af09ad5a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx\" (UID: \"7a33ad9e-488b-4002-88b0-18af09ad5a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:08.605806 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.605781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a33ad9e-488b-4002-88b0-18af09ad5a82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx\" (UID: \"7a33ad9e-488b-4002-88b0-18af09ad5a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:08.612390 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.612360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8r5w\" (UniqueName: \"kubernetes.io/projected/7a33ad9e-488b-4002-88b0-18af09ad5a82-kube-api-access-c8r5w\") pod \"payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx\" (UID: \"7a33ad9e-488b-4002-88b0-18af09ad5a82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:08.705704 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.705631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae815d4d-cf31-4a5a-9705-a218990a4bb5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw\" (UID: \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:08.705891 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.705727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nvc\" (UniqueName: \"kubernetes.io/projected/ae815d4d-cf31-4a5a-9705-a218990a4bb5-kube-api-access-r9nvc\") pod \"payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw\" (UID: \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:08.706349 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.706327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae815d4d-cf31-4a5a-9705-a218990a4bb5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw\" (UID: \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:08.713404 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.713374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nvc\" (UniqueName: \"kubernetes.io/projected/ae815d4d-cf31-4a5a-9705-a218990a4bb5-kube-api-access-r9nvc\") pod \"payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw\" (UID: \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:08.800397 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.800301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:08.811753 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:08.811728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:09.080391 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.080357    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"]
Sep 11 22:23:09.084802 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:23:09.084759    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a33ad9e_488b_4002_88b0_18af09ad5a82.slice/crio-5fb9f13ec86ae20b729a43fc09e95881fb61e3ec795f0d18395260daae144222 WatchSource:0}: Error finding container 5fb9f13ec86ae20b729a43fc09e95881fb61e3ec795f0d18395260daae144222: Status 404 returned error can't find the container with id 5fb9f13ec86ae20b729a43fc09e95881fb61e3ec795f0d18395260daae144222
Sep 11 22:23:09.099357 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.099332    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"]
Sep 11 22:23:09.100621 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:23:09.100595    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae815d4d_cf31_4a5a_9705_a218990a4bb5.slice/crio-2765cc565fe477253bc515e064ae303dcb183e045121ce2718e525d2aca1fdb8 WatchSource:0}: Error finding container 2765cc565fe477253bc515e064ae303dcb183e045121ce2718e525d2aca1fdb8: Status 404 returned error can't find the container with id 2765cc565fe477253bc515e064ae303dcb183e045121ce2718e525d2aca1fdb8
Sep 11 22:23:09.555010 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.554973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx" event={"ID":"7a33ad9e-488b-4002-88b0-18af09ad5a82","Type":"ContainerStarted","Data":"9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850"}
Sep 11 22:23:09.555763 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.555058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx" event={"ID":"7a33ad9e-488b-4002-88b0-18af09ad5a82","Type":"ContainerStarted","Data":"5fb9f13ec86ae20b729a43fc09e95881fb61e3ec795f0d18395260daae144222"}
Sep 11 22:23:09.555763 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.555133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:09.556782 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.556759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw" event={"ID":"ae815d4d-cf31-4a5a-9705-a218990a4bb5","Type":"ContainerStarted","Data":"1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2"}
Sep 11 22:23:09.556859 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.556799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw" event={"ID":"ae815d4d-cf31-4a5a-9705-a218990a4bb5","Type":"ContainerStarted","Data":"2765cc565fe477253bc515e064ae303dcb183e045121ce2718e525d2aca1fdb8"}
Sep 11 22:23:09.556925 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.556885    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:09.571197 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.571143    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx" podStartSLOduration=1.571124873 podStartE2EDuration="1.571124873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:23:09.568862467 +0000 UTC m=+1851.253809183" watchObservedRunningTime="2026-09-11 22:23:09.571124873 +0000 UTC m=+1851.256071591"
Sep 11 22:23:09.585795 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:09.585747    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw" podStartSLOduration=1.585732262 podStartE2EDuration="1.585732262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:23:09.582916791 +0000 UTC m=+1851.267863498" watchObservedRunningTime="2026-09-11 22:23:09.585732262 +0000 UTC m=+1851.270678976"
Sep 11 22:23:19.255465 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:19.255429    2578 scope.go:117] "RemoveContainer" containerID="d5aed3b9ac09e01a9bebcf993e0ce35be55d4cf2b74b67ecfab2fdc65086fcac"
Sep 11 22:23:20.564833 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:20.564803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:20.565271 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:20.565040    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:35.242353 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.242314    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"]
Sep 11 22:23:35.250211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.250173    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"
Sep 11 22:23:35.252685 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.252656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:23:35.254456 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.254418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"]
Sep 11 22:23:35.358652 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.358597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163edae6-082f-402d-94ac-62b56348981e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt\" (UID: \"163edae6-082f-402d-94ac-62b56348981e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"
Sep 11 22:23:35.459603 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.459563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163edae6-082f-402d-94ac-62b56348981e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt\" (UID: \"163edae6-082f-402d-94ac-62b56348981e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"
Sep 11 22:23:35.460225 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.460203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163edae6-082f-402d-94ac-62b56348981e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt\" (UID: \"163edae6-082f-402d-94ac-62b56348981e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"
Sep 11 22:23:35.564618 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.564534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"
Sep 11 22:23:35.833806 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:35.833779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"]
Sep 11 22:23:35.835953 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:23:35.835921    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163edae6_082f_402d_94ac_62b56348981e.slice/crio-b6460c799166f82dd74458f9cb4e6979a1953d1c39d8b2c4a69557a02b47bd8d WatchSource:0}: Error finding container b6460c799166f82dd74458f9cb4e6979a1953d1c39d8b2c4a69557a02b47bd8d: Status 404 returned error can't find the container with id b6460c799166f82dd74458f9cb4e6979a1953d1c39d8b2c4a69557a02b47bd8d
Sep 11 22:23:36.719683 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:36.719620    2578 generic.go:356] "Generic (PLEG): container finished" podID="163edae6-082f-402d-94ac-62b56348981e" containerID="f66134e7fa3b9bfd59299a675b5878ab786b6a0a02d7bad25e82a756c84ff6c1" exitCode=0
Sep 11 22:23:36.720067 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:36.719702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt" event={"ID":"163edae6-082f-402d-94ac-62b56348981e","Type":"ContainerDied","Data":"f66134e7fa3b9bfd59299a675b5878ab786b6a0a02d7bad25e82a756c84ff6c1"}
Sep 11 22:23:36.720067 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:36.719755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt" event={"ID":"163edae6-082f-402d-94ac-62b56348981e","Type":"ContainerStarted","Data":"b6460c799166f82dd74458f9cb4e6979a1953d1c39d8b2c4a69557a02b47bd8d"}
Sep 11 22:23:37.856428 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:37.856400    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"
Sep 11 22:23:37.984722 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:37.984615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163edae6-082f-402d-94ac-62b56348981e-maas-api-service-ca\") pod \"163edae6-082f-402d-94ac-62b56348981e\" (UID: \"163edae6-082f-402d-94ac-62b56348981e\") "
Sep 11 22:23:37.985047 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:37.985025    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163edae6-082f-402d-94ac-62b56348981e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "163edae6-082f-402d-94ac-62b56348981e" (UID: "163edae6-082f-402d-94ac-62b56348981e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:23:38.085668 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:38.085626    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163edae6-082f-402d-94ac-62b56348981e-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:23:38.731799 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:38.731764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt" event={"ID":"163edae6-082f-402d-94ac-62b56348981e","Type":"ContainerDied","Data":"b6460c799166f82dd74458f9cb4e6979a1953d1c39d8b2c4a69557a02b47bd8d"}
Sep 11 22:23:38.731978 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:38.731803    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"
Sep 11 22:23:38.731978 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:38.731813    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6460c799166f82dd74458f9cb4e6979a1953d1c39d8b2c4a69557a02b47bd8d"
Sep 11 22:23:46.180625 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.180531    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"]
Sep 11 22:23:46.181435 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.181376    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw" podUID="ae815d4d-cf31-4a5a-9705-a218990a4bb5" containerName="payload-processing" containerID="cri-o://1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2" gracePeriod=30
Sep 11 22:23:46.184292 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.184266    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"]
Sep 11 22:23:46.184477 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.184455    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx" podUID="7a33ad9e-488b-4002-88b0-18af09ad5a82" containerName="payload-pre-processing" containerID="cri-o://9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850" gracePeriod=30
Sep 11 22:23:46.335745 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.335698    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"]
Sep 11 22:23:46.338518 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.338478    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1688a8-wgpqt"]
Sep 11 22:23:46.458083 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.458059    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:46.460955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.460936    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:46.555956 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.555922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a33ad9e-488b-4002-88b0-18af09ad5a82-plugins-config-volume\") pod \"7a33ad9e-488b-4002-88b0-18af09ad5a82\" (UID: \"7a33ad9e-488b-4002-88b0-18af09ad5a82\") "
Sep 11 22:23:46.556151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.555989    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9nvc\" (UniqueName: \"kubernetes.io/projected/ae815d4d-cf31-4a5a-9705-a218990a4bb5-kube-api-access-r9nvc\") pod \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\" (UID: \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\") "
Sep 11 22:23:46.556151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.556011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae815d4d-cf31-4a5a-9705-a218990a4bb5-plugins-config-volume\") pod \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\" (UID: \"ae815d4d-cf31-4a5a-9705-a218990a4bb5\") "
Sep 11 22:23:46.556151 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.556047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8r5w\" (UniqueName: \"kubernetes.io/projected/7a33ad9e-488b-4002-88b0-18af09ad5a82-kube-api-access-c8r5w\") pod \"7a33ad9e-488b-4002-88b0-18af09ad5a82\" (UID: \"7a33ad9e-488b-4002-88b0-18af09ad5a82\") "
Sep 11 22:23:46.556396 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.556369    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a33ad9e-488b-4002-88b0-18af09ad5a82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a33ad9e-488b-4002-88b0-18af09ad5a82" (UID: "7a33ad9e-488b-4002-88b0-18af09ad5a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:23:46.556449 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.556426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae815d4d-cf31-4a5a-9705-a218990a4bb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae815d4d-cf31-4a5a-9705-a218990a4bb5" (UID: "ae815d4d-cf31-4a5a-9705-a218990a4bb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:23:46.558223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.558199    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a33ad9e-488b-4002-88b0-18af09ad5a82-kube-api-access-c8r5w" (OuterVolumeSpecName: "kube-api-access-c8r5w") pod "7a33ad9e-488b-4002-88b0-18af09ad5a82" (UID: "7a33ad9e-488b-4002-88b0-18af09ad5a82"). InnerVolumeSpecName "kube-api-access-c8r5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:23:46.558223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.558211    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae815d4d-cf31-4a5a-9705-a218990a4bb5-kube-api-access-r9nvc" (OuterVolumeSpecName: "kube-api-access-r9nvc") pod "ae815d4d-cf31-4a5a-9705-a218990a4bb5" (UID: "ae815d4d-cf31-4a5a-9705-a218990a4bb5"). InnerVolumeSpecName "kube-api-access-r9nvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:23:46.656802 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.656764    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9nvc\" (UniqueName: \"kubernetes.io/projected/ae815d4d-cf31-4a5a-9705-a218990a4bb5-kube-api-access-r9nvc\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:23:46.656802 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.656796    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae815d4d-cf31-4a5a-9705-a218990a4bb5-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:23:46.656802 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.656805    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8r5w\" (UniqueName: \"kubernetes.io/projected/7a33ad9e-488b-4002-88b0-18af09ad5a82-kube-api-access-c8r5w\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:23:46.657047 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.656814    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a33ad9e-488b-4002-88b0-18af09ad5a82-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:23:46.785550 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.785454    2578 generic.go:356] "Generic (PLEG): container finished" podID="ae815d4d-cf31-4a5a-9705-a218990a4bb5" containerID="1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2" exitCode=0
Sep 11 22:23:46.785550 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.785509    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"
Sep 11 22:23:46.785550 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.785530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw" event={"ID":"ae815d4d-cf31-4a5a-9705-a218990a4bb5","Type":"ContainerDied","Data":"1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2"}
Sep 11 22:23:46.785858 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.785602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw" event={"ID":"ae815d4d-cf31-4a5a-9705-a218990a4bb5","Type":"ContainerDied","Data":"2765cc565fe477253bc515e064ae303dcb183e045121ce2718e525d2aca1fdb8"}
Sep 11 22:23:46.785858 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.785625    2578 scope.go:117] "RemoveContainer" containerID="1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2"
Sep 11 22:23:46.787140 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.787111    2578 generic.go:356] "Generic (PLEG): container finished" podID="7a33ad9e-488b-4002-88b0-18af09ad5a82" containerID="9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850" exitCode=0
Sep 11 22:23:46.787273 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.787170    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"
Sep 11 22:23:46.787273 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.787191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx" event={"ID":"7a33ad9e-488b-4002-88b0-18af09ad5a82","Type":"ContainerDied","Data":"9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850"}
Sep 11 22:23:46.787273 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.787223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx" event={"ID":"7a33ad9e-488b-4002-88b0-18af09ad5a82","Type":"ContainerDied","Data":"5fb9f13ec86ae20b729a43fc09e95881fb61e3ec795f0d18395260daae144222"}
Sep 11 22:23:46.798758 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.798508    2578 scope.go:117] "RemoveContainer" containerID="1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2"
Sep 11 22:23:46.799036 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:23:46.799014    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2\": container with ID starting with 1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2 not found: ID does not exist" containerID="1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2"
Sep 11 22:23:46.799135 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.799041    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2"} err="failed to get container status \"1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2\": rpc error: code = NotFound desc = could not find container \"1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2\": container with ID starting with 1842dc74f51d9d7d25519783611b8bc53d65ea5ea435a2dd074ac9afc0c26de2 not found: ID does not exist"
Sep 11 22:23:46.799135 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.799059    2578 scope.go:117] "RemoveContainer" containerID="9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850"
Sep 11 22:23:46.807704 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.807680    2578 scope.go:117] "RemoveContainer" containerID="9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850"
Sep 11 22:23:46.807989 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:23:46.807971    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850\": container with ID starting with 9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850 not found: ID does not exist" containerID="9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850"
Sep 11 22:23:46.808041 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.807997    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850"} err="failed to get container status \"9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850\": rpc error: code = NotFound desc = could not find container \"9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850\": container with ID starting with 9e2c73ecc642e7b7a2f8f9dcbe14f89ba35c42ce9131ee0dcc5d0f4f23e7b850 not found: ID does not exist"
Sep 11 22:23:46.812033 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.812014    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"]
Sep 11 22:23:46.814251 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.814232    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1688a8-6bddc5d8f8-ljmfx"]
Sep 11 22:23:46.826463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.826435    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"]
Sep 11 22:23:46.829559 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.829538    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1688a8-54dd58b47f-djhlw"]
Sep 11 22:23:46.858518 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.858476    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163edae6-082f-402d-94ac-62b56348981e" path="/var/lib/kubelet/pods/163edae6-082f-402d-94ac-62b56348981e/volumes"
Sep 11 22:23:46.858883 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.858869    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a33ad9e-488b-4002-88b0-18af09ad5a82" path="/var/lib/kubelet/pods/7a33ad9e-488b-4002-88b0-18af09ad5a82/volumes"
Sep 11 22:23:46.859195 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:46.859184    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae815d4d-cf31-4a5a-9705-a218990a4bb5" path="/var/lib/kubelet/pods/ae815d4d-cf31-4a5a-9705-a218990a4bb5/volumes"
Sep 11 22:23:56.289389 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289351    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"]
Sep 11 22:23:56.289820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289729    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a33ad9e-488b-4002-88b0-18af09ad5a82" containerName="payload-pre-processing"
Sep 11 22:23:56.289820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289742    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a33ad9e-488b-4002-88b0-18af09ad5a82" containerName="payload-pre-processing"
Sep 11 22:23:56.289820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289759    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae815d4d-cf31-4a5a-9705-a218990a4bb5" containerName="payload-processing"
Sep 11 22:23:56.289820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289765    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae815d4d-cf31-4a5a-9705-a218990a4bb5" containerName="payload-processing"
Sep 11 22:23:56.289820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289778    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163edae6-082f-402d-94ac-62b56348981e" containerName="revoke-keys"
Sep 11 22:23:56.289820 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289784    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="163edae6-082f-402d-94ac-62b56348981e" containerName="revoke-keys"
Sep 11 22:23:56.290032 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289838    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a33ad9e-488b-4002-88b0-18af09ad5a82" containerName="payload-pre-processing"
Sep 11 22:23:56.290032 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289848    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae815d4d-cf31-4a5a-9705-a218990a4bb5" containerName="payload-processing"
Sep 11 22:23:56.290032 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.289856    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="163edae6-082f-402d-94ac-62b56348981e" containerName="revoke-keys"
Sep 11 22:23:56.292900 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.292882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"
Sep 11 22:23:56.294946 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.294915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:23:56.298715 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.298692    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"]
Sep 11 22:23:56.341785 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.341745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72bb99a-c371-4546-885f-60667eba9ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg\" (UID: \"b72bb99a-c371-4546-885f-60667eba9ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"
Sep 11 22:23:56.442623 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.442584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72bb99a-c371-4546-885f-60667eba9ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg\" (UID: \"b72bb99a-c371-4546-885f-60667eba9ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"
Sep 11 22:23:56.443242 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.443220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72bb99a-c371-4546-885f-60667eba9ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg\" (UID: \"b72bb99a-c371-4546-885f-60667eba9ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"
Sep 11 22:23:56.605999 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.605906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"
Sep 11 22:23:56.878197 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:56.878166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"]
Sep 11 22:23:56.880076 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:23:56.880047    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72bb99a_c371_4546_885f_60667eba9ee1.slice/crio-2284dc28b1b594d35c8c9cc21911ea959f493122a17f83e9605d43452c423c62 WatchSource:0}: Error finding container 2284dc28b1b594d35c8c9cc21911ea959f493122a17f83e9605d43452c423c62: Status 404 returned error can't find the container with id 2284dc28b1b594d35c8c9cc21911ea959f493122a17f83e9605d43452c423c62
Sep 11 22:23:57.854280 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:57.854185    2578 generic.go:356] "Generic (PLEG): container finished" podID="b72bb99a-c371-4546-885f-60667eba9ee1" containerID="c0bcf3c82342646027c7ab4576107762d4cc19c9a865483dfae2afdd31088f13" exitCode=0
Sep 11 22:23:57.854280 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:57.854238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg" event={"ID":"b72bb99a-c371-4546-885f-60667eba9ee1","Type":"ContainerDied","Data":"c0bcf3c82342646027c7ab4576107762d4cc19c9a865483dfae2afdd31088f13"}
Sep 11 22:23:57.854740 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:57.854283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg" event={"ID":"b72bb99a-c371-4546-885f-60667eba9ee1","Type":"ContainerStarted","Data":"2284dc28b1b594d35c8c9cc21911ea959f493122a17f83e9605d43452c423c62"}
Sep 11 22:23:58.991840 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:58.991815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"
Sep 11 22:23:59.067520 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:59.067485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72bb99a-c371-4546-885f-60667eba9ee1-maas-api-service-ca\") pod \"b72bb99a-c371-4546-885f-60667eba9ee1\" (UID: \"b72bb99a-c371-4546-885f-60667eba9ee1\") "
Sep 11 22:23:59.067903 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:59.067879    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72bb99a-c371-4546-885f-60667eba9ee1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b72bb99a-c371-4546-885f-60667eba9ee1" (UID: "b72bb99a-c371-4546-885f-60667eba9ee1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:23:59.168478 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:59.168448    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72bb99a-c371-4546-885f-60667eba9ee1-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:23:59.866520 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:59.866480    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"
Sep 11 22:23:59.866743 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:59.866526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg" event={"ID":"b72bb99a-c371-4546-885f-60667eba9ee1","Type":"ContainerDied","Data":"2284dc28b1b594d35c8c9cc21911ea959f493122a17f83e9605d43452c423c62"}
Sep 11 22:23:59.866743 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:23:59.866575    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2284dc28b1b594d35c8c9cc21911ea959f493122a17f83e9605d43452c423c62"
Sep 11 22:24:12.151933 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:12.151898    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"]
Sep 11 22:24:12.155324 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:12.155297    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6fe4c2-pzzvg"]
Sep 11 22:24:12.857128 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:12.857093    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72bb99a-c371-4546-885f-60667eba9ee1" path="/var/lib/kubelet/pods/b72bb99a-c371-4546-885f-60667eba9ee1/volumes"
Sep 11 22:24:17.516770 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:17.516729    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"]
Sep 11 22:24:19.278219 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:19.278185    2578 scope.go:117] "RemoveContainer" containerID="b51040932a715847b3744f7f7020b777870e6cbf77cb99dc6368a74ccb60b1d7"
Sep 11 22:24:19.287262 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:19.287236    2578 scope.go:117] "RemoveContainer" containerID="fe81b2efc5b100a961db62a0a8f3f5174c74aab328beb4fb33df1dd39e647b06"
Sep 11 22:24:19.295507 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:19.295486    2578 scope.go:117] "RemoveContainer" containerID="f55f81d847c8cbb3ad2d23e4f103f8f5550b1ff9491cef7d321d26b746b478fa"
Sep 11 22:24:19.303439 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:19.303410    2578 scope.go:117] "RemoveContainer" containerID="e1f9ff0086963b1e4c752fe4e746f0fc886b40eeeaf1875f821924253e20a3c9"
Sep 11 22:24:32.536522 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.536468    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5" podUID="899ecd8c-27ca-4af6-8fb4-ac66419f95c0" containerName="main" containerID="cri-o://9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568" gracePeriod=45
Sep 11 22:24:32.789374 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.789313    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:24:32.893387 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-model-cache\") pod \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") "
Sep 11 22:24:32.893387 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmkph\" (UniqueName: \"kubernetes.io/projected/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-kube-api-access-jmkph\") pod \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") "
Sep 11 22:24:32.893588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tmp-dir\") pod \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") "
Sep 11 22:24:32.893588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-home\") pod \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") "
Sep 11 22:24:32.893588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-dshm\") pod \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") "
Sep 11 22:24:32.893588 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tls-certs\") pod \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\" (UID: \"899ecd8c-27ca-4af6-8fb4-ac66419f95c0\") "
Sep 11 22:24:32.893822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "899ecd8c-27ca-4af6-8fb4-ac66419f95c0" (UID: "899ecd8c-27ca-4af6-8fb4-ac66419f95c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:24:32.893822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893767    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-home" (OuterVolumeSpecName: "home") pod "899ecd8c-27ca-4af6-8fb4-ac66419f95c0" (UID: "899ecd8c-27ca-4af6-8fb4-ac66419f95c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:24:32.893822 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "899ecd8c-27ca-4af6-8fb4-ac66419f95c0" (UID: "899ecd8c-27ca-4af6-8fb4-ac66419f95c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:24:32.894000 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.893986    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:24:32.894056 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.894007    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:24:32.894056 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.894022    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:24:32.895875 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.895845    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-dshm" (OuterVolumeSpecName: "dshm") pod "899ecd8c-27ca-4af6-8fb4-ac66419f95c0" (UID: "899ecd8c-27ca-4af6-8fb4-ac66419f95c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:24:32.895875 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.895854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "899ecd8c-27ca-4af6-8fb4-ac66419f95c0" (UID: "899ecd8c-27ca-4af6-8fb4-ac66419f95c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:24:32.896032 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.895909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-kube-api-access-jmkph" (OuterVolumeSpecName: "kube-api-access-jmkph") pod "899ecd8c-27ca-4af6-8fb4-ac66419f95c0" (UID: "899ecd8c-27ca-4af6-8fb4-ac66419f95c0"). InnerVolumeSpecName "kube-api-access-jmkph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:24:32.995341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.995298    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:24:32.995341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.995333    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmkph\" (UniqueName: \"kubernetes.io/projected/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-kube-api-access-jmkph\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:24:32.995341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:32.995349    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899ecd8c-27ca-4af6-8fb4-ac66419f95c0-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:24:33.056300 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.056202    2578 generic.go:356] "Generic (PLEG): container finished" podID="899ecd8c-27ca-4af6-8fb4-ac66419f95c0" containerID="9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568" exitCode=0
Sep 11 22:24:33.056300 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.056247    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"
Sep 11 22:24:33.056521 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.056292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5" event={"ID":"899ecd8c-27ca-4af6-8fb4-ac66419f95c0","Type":"ContainerDied","Data":"9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568"}
Sep 11 22:24:33.056521 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.056351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5" event={"ID":"899ecd8c-27ca-4af6-8fb4-ac66419f95c0","Type":"ContainerDied","Data":"8c6d76cfd8b22e2291b47624e13fb2906dc2cb5e87f277d0de335564790a88a8"}
Sep 11 22:24:33.056521 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.056375    2578 scope.go:117] "RemoveContainer" containerID="9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568"
Sep 11 22:24:33.065918 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.065900    2578 scope.go:117] "RemoveContainer" containerID="9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568"
Sep 11 22:24:33.066201 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:24:33.066178    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568\": container with ID starting with 9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568 not found: ID does not exist" containerID="9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568"
Sep 11 22:24:33.066271 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.066211    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568"} err="failed to get container status \"9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568\": rpc error: code = NotFound desc = could not find container \"9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568\": container with ID starting with 9c69e59c8fb246289f4a0a744d71bb947fadbc0b35fe74d96e68ba63f9ff0568 not found: ID does not exist"
Sep 11 22:24:33.077160 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.077134    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"]
Sep 11 22:24:33.080299 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:33.080278    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6fe4c2/ipp-route-6fe4c2-kserve-84b5d69579-cz2h5"]
Sep 11 22:24:34.857121 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:24:34.857084    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899ecd8c-27ca-4af6-8fb4-ac66419f95c0" path="/var/lib/kubelet/pods/899ecd8c-27ca-4af6-8fb4-ac66419f95c0/volumes"
Sep 11 22:25:00.893958 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.893921    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"]
Sep 11 22:25:00.894370 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.894272    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899ecd8c-27ca-4af6-8fb4-ac66419f95c0" containerName="main"
Sep 11 22:25:00.894370 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.894284    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="899ecd8c-27ca-4af6-8fb4-ac66419f95c0" containerName="main"
Sep 11 22:25:00.894370 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.894294    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72bb99a-c371-4546-885f-60667eba9ee1" containerName="revoke-keys"
Sep 11 22:25:00.894370 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.894299    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72bb99a-c371-4546-885f-60667eba9ee1" containerName="revoke-keys"
Sep 11 22:25:00.894370 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.894368    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="899ecd8c-27ca-4af6-8fb4-ac66419f95c0" containerName="main"
Sep 11 22:25:00.894544 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.894377    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72bb99a-c371-4546-885f-60667eba9ee1" containerName="revoke-keys"
Sep 11 22:25:00.897584 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.897563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"
Sep 11 22:25:00.899624 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.899604    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:25:00.905358 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:00.905219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"]
Sep 11 22:25:01.056276 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:01.056234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d83a84-c63a-4c6c-9c26-150321a68ff5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb\" (UID: \"73d83a84-c63a-4c6c-9c26-150321a68ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"
Sep 11 22:25:01.157303 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:01.157198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d83a84-c63a-4c6c-9c26-150321a68ff5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb\" (UID: \"73d83a84-c63a-4c6c-9c26-150321a68ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"
Sep 11 22:25:01.157868 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:01.157847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d83a84-c63a-4c6c-9c26-150321a68ff5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb\" (UID: \"73d83a84-c63a-4c6c-9c26-150321a68ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"
Sep 11 22:25:01.209302 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:01.209260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"
Sep 11 22:25:01.480532 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:01.480507    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"]
Sep 11 22:25:01.482680 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:25:01.482653    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d83a84_c63a_4c6c_9c26_150321a68ff5.slice/crio-1992555a838d367c9c6d2b8dfcce569a058fa7f45a7cf0f24f7c207c955ab95f WatchSource:0}: Error finding container 1992555a838d367c9c6d2b8dfcce569a058fa7f45a7cf0f24f7c207c955ab95f: Status 404 returned error can't find the container with id 1992555a838d367c9c6d2b8dfcce569a058fa7f45a7cf0f24f7c207c955ab95f
Sep 11 22:25:02.217593 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:02.217555    2578 generic.go:356] "Generic (PLEG): container finished" podID="73d83a84-c63a-4c6c-9c26-150321a68ff5" containerID="15917603273ef4409a0815f43dc680990a1f79ab6c01d7b850e7ea12d176020c" exitCode=0
Sep 11 22:25:02.218025 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:02.217672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb" event={"ID":"73d83a84-c63a-4c6c-9c26-150321a68ff5","Type":"ContainerDied","Data":"15917603273ef4409a0815f43dc680990a1f79ab6c01d7b850e7ea12d176020c"}
Sep 11 22:25:02.218025 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:02.217740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb" event={"ID":"73d83a84-c63a-4c6c-9c26-150321a68ff5","Type":"ContainerStarted","Data":"1992555a838d367c9c6d2b8dfcce569a058fa7f45a7cf0f24f7c207c955ab95f"}
Sep 11 22:25:03.351742 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:03.351719    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"
Sep 11 22:25:03.481191 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:03.481101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d83a84-c63a-4c6c-9c26-150321a68ff5-maas-api-service-ca\") pod \"73d83a84-c63a-4c6c-9c26-150321a68ff5\" (UID: \"73d83a84-c63a-4c6c-9c26-150321a68ff5\") "
Sep 11 22:25:03.481472 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:03.481447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d83a84-c63a-4c6c-9c26-150321a68ff5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73d83a84-c63a-4c6c-9c26-150321a68ff5" (UID: "73d83a84-c63a-4c6c-9c26-150321a68ff5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:03.582564 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:03.582528    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d83a84-c63a-4c6c-9c26-150321a68ff5-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:04.229610 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:04.229567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb" event={"ID":"73d83a84-c63a-4c6c-9c26-150321a68ff5","Type":"ContainerDied","Data":"1992555a838d367c9c6d2b8dfcce569a058fa7f45a7cf0f24f7c207c955ab95f"}
Sep 11 22:25:04.229793 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:04.229630    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1992555a838d367c9c6d2b8dfcce569a058fa7f45a7cf0f24f7c207c955ab95f"
Sep 11 22:25:04.229793 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:04.229594    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"
Sep 11 22:25:11.395240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.395158    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"]
Sep 11 22:25:11.395711 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.395389    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm" podUID="55a7eaed-ca97-48b5-8878-a33984e8e6e8" containerName="payload-pre-processing" containerID="cri-o://2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a" gracePeriod=30
Sep 11 22:25:11.566819 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.566788    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"]
Sep 11 22:25:11.569411 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.569389    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dbaf52-ghbkb"]
Sep 11 22:25:11.639531 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.639502    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:25:11.660416 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.660348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a7eaed-ca97-48b5-8878-a33984e8e6e8-plugins-config-volume\") pod \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\" (UID: \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\") "
Sep 11 22:25:11.660416 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.660397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2ndk\" (UniqueName: \"kubernetes.io/projected/55a7eaed-ca97-48b5-8878-a33984e8e6e8-kube-api-access-r2ndk\") pod \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\" (UID: \"55a7eaed-ca97-48b5-8878-a33984e8e6e8\") "
Sep 11 22:25:11.660825 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.660800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a7eaed-ca97-48b5-8878-a33984e8e6e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55a7eaed-ca97-48b5-8878-a33984e8e6e8" (UID: "55a7eaed-ca97-48b5-8878-a33984e8e6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:11.662616 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.662592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a7eaed-ca97-48b5-8878-a33984e8e6e8-kube-api-access-r2ndk" (OuterVolumeSpecName: "kube-api-access-r2ndk") pod "55a7eaed-ca97-48b5-8878-a33984e8e6e8" (UID: "55a7eaed-ca97-48b5-8878-a33984e8e6e8"). InnerVolumeSpecName "kube-api-access-r2ndk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:11.761235 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.761197    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a7eaed-ca97-48b5-8878-a33984e8e6e8-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:11.761235 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:11.761225    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2ndk\" (UniqueName: \"kubernetes.io/projected/55a7eaed-ca97-48b5-8878-a33984e8e6e8-kube-api-access-r2ndk\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:12.279652 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.279612    2578 generic.go:356] "Generic (PLEG): container finished" podID="55a7eaed-ca97-48b5-8878-a33984e8e6e8" containerID="2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a" exitCode=0
Sep 11 22:25:12.279856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.279685    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"
Sep 11 22:25:12.279856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.279678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm" event={"ID":"55a7eaed-ca97-48b5-8878-a33984e8e6e8","Type":"ContainerDied","Data":"2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a"}
Sep 11 22:25:12.279856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.279809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm" event={"ID":"55a7eaed-ca97-48b5-8878-a33984e8e6e8","Type":"ContainerDied","Data":"3e72315f10cea9645b9b696d6d727c715f0810ad89ed38685016b386e04d05c1"}
Sep 11 22:25:12.279856 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.279833    2578 scope.go:117] "RemoveContainer" containerID="2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a"
Sep 11 22:25:12.289241 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.289222    2578 scope.go:117] "RemoveContainer" containerID="2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a"
Sep 11 22:25:12.289534 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:25:12.289513    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a\": container with ID starting with 2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a not found: ID does not exist" containerID="2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a"
Sep 11 22:25:12.289585 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.289544    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a"} err="failed to get container status \"2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a\": rpc error: code = NotFound desc = could not find container \"2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a\": container with ID starting with 2720f221f64f08d9be8d6db10a4642502a3c0f7920c35aa56f81bf46a6ecbb7a not found: ID does not exist"
Sep 11 22:25:12.300506 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.300480    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"]
Sep 11 22:25:12.304956 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.304931    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbaf52-f4544494b-zphnm"]
Sep 11 22:25:12.857161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.857128    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a7eaed-ca97-48b5-8878-a33984e8e6e8" path="/var/lib/kubelet/pods/55a7eaed-ca97-48b5-8878-a33984e8e6e8/volumes"
Sep 11 22:25:12.857543 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:12.857503    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73d83a84-c63a-4c6c-9c26-150321a68ff5" path="/var/lib/kubelet/pods/73d83a84-c63a-4c6c-9c26-150321a68ff5/volumes"
Sep 11 22:25:19.347419 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:19.347387    2578 scope.go:117] "RemoveContainer" containerID="67b9223c70e79477a7cd3c54d85ee1e4d43240cfef7d0a69a0cf6dc49d403434"
Sep 11 22:25:29.156080 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.156048    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"]
Sep 11 22:25:29.159861 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.159827    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d09c96-29819415-hcpng"]
Sep 11 22:25:29.179309 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.179269    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"]
Sep 11 22:25:29.179599 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.179546    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw" podUID="30854b6d-6fc1-4d3b-8e27-edef38c380a7" containerName="payload-processing" containerID="cri-o://1ee703290fd4e76570095b80a99560b1e233f4ee0f6aa14e3224a8e0abd95023" gracePeriod=30
Sep 11 22:25:29.186036 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.185997    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"]
Sep 11 22:25:29.186254 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.186216    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2" podUID="54933a8e-de7f-44a9-9c55-b21e3ece54a8" containerName="payload-pre-processing" containerID="cri-o://f2a56967c68d3f7adc460417866107ccf098797b758e90600296ecedc0ddf81c" gracePeriod=30
Sep 11 22:25:29.376745 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.376708    2578 generic.go:356] "Generic (PLEG): container finished" podID="30854b6d-6fc1-4d3b-8e27-edef38c380a7" containerID="1ee703290fd4e76570095b80a99560b1e233f4ee0f6aa14e3224a8e0abd95023" exitCode=0
Sep 11 22:25:29.376926 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.376793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw" event={"ID":"30854b6d-6fc1-4d3b-8e27-edef38c380a7","Type":"ContainerDied","Data":"1ee703290fd4e76570095b80a99560b1e233f4ee0f6aa14e3224a8e0abd95023"}
Sep 11 22:25:29.378721 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.378698    2578 generic.go:356] "Generic (PLEG): container finished" podID="54933a8e-de7f-44a9-9c55-b21e3ece54a8" containerID="f2a56967c68d3f7adc460417866107ccf098797b758e90600296ecedc0ddf81c" exitCode=0
Sep 11 22:25:29.378844 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.378774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2" event={"ID":"54933a8e-de7f-44a9-9c55-b21e3ece54a8","Type":"ContainerDied","Data":"f2a56967c68d3f7adc460417866107ccf098797b758e90600296ecedc0ddf81c"}
Sep 11 22:25:29.457921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.457895    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:25:29.460961 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.460940    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:25:29.520010 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.519958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjqnm\" (UniqueName: \"kubernetes.io/projected/54933a8e-de7f-44a9-9c55-b21e3ece54a8-kube-api-access-cjqnm\") pod \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\" (UID: \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\") "
Sep 11 22:25:29.520218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.520075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xk5t\" (UniqueName: \"kubernetes.io/projected/30854b6d-6fc1-4d3b-8e27-edef38c380a7-kube-api-access-2xk5t\") pod \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\" (UID: \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\") "
Sep 11 22:25:29.520218 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.520166    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30854b6d-6fc1-4d3b-8e27-edef38c380a7-plugins-config-volume\") pod \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\" (UID: \"30854b6d-6fc1-4d3b-8e27-edef38c380a7\") "
Sep 11 22:25:29.520341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.520259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54933a8e-de7f-44a9-9c55-b21e3ece54a8-plugins-config-volume\") pod \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\" (UID: \"54933a8e-de7f-44a9-9c55-b21e3ece54a8\") "
Sep 11 22:25:29.520592 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.520566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30854b6d-6fc1-4d3b-8e27-edef38c380a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30854b6d-6fc1-4d3b-8e27-edef38c380a7" (UID: "30854b6d-6fc1-4d3b-8e27-edef38c380a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:29.520732 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.520617    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54933a8e-de7f-44a9-9c55-b21e3ece54a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54933a8e-de7f-44a9-9c55-b21e3ece54a8" (UID: "54933a8e-de7f-44a9-9c55-b21e3ece54a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:29.522791 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.522761    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30854b6d-6fc1-4d3b-8e27-edef38c380a7-kube-api-access-2xk5t" (OuterVolumeSpecName: "kube-api-access-2xk5t") pod "30854b6d-6fc1-4d3b-8e27-edef38c380a7" (UID: "30854b6d-6fc1-4d3b-8e27-edef38c380a7"). InnerVolumeSpecName "kube-api-access-2xk5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:29.522914 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.522813    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54933a8e-de7f-44a9-9c55-b21e3ece54a8-kube-api-access-cjqnm" (OuterVolumeSpecName: "kube-api-access-cjqnm") pod "54933a8e-de7f-44a9-9c55-b21e3ece54a8" (UID: "54933a8e-de7f-44a9-9c55-b21e3ece54a8"). InnerVolumeSpecName "kube-api-access-cjqnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:29.621223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.621186    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjqnm\" (UniqueName: \"kubernetes.io/projected/54933a8e-de7f-44a9-9c55-b21e3ece54a8-kube-api-access-cjqnm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:29.621223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.621218    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xk5t\" (UniqueName: \"kubernetes.io/projected/30854b6d-6fc1-4d3b-8e27-edef38c380a7-kube-api-access-2xk5t\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:29.621223 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.621227    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30854b6d-6fc1-4d3b-8e27-edef38c380a7-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:29.621474 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:29.621238    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54933a8e-de7f-44a9-9c55-b21e3ece54a8-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:30.385783 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.385754    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"
Sep 11 22:25:30.386237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.385752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw" event={"ID":"30854b6d-6fc1-4d3b-8e27-edef38c380a7","Type":"ContainerDied","Data":"a826195a942ddc6cd46d318120689d52be8f49101b8360020de887c98d76e426"}
Sep 11 22:25:30.386237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.385894    2578 scope.go:117] "RemoveContainer" containerID="1ee703290fd4e76570095b80a99560b1e233f4ee0f6aa14e3224a8e0abd95023"
Sep 11 22:25:30.387293 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.387212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2" event={"ID":"54933a8e-de7f-44a9-9c55-b21e3ece54a8","Type":"ContainerDied","Data":"330f1f750424c553683db00b6fe0f52e111c945e6865e7975312320431d92c49"}
Sep 11 22:25:30.387293 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.387245    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"
Sep 11 22:25:30.396905 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.396603    2578 scope.go:117] "RemoveContainer" containerID="f2a56967c68d3f7adc460417866107ccf098797b758e90600296ecedc0ddf81c"
Sep 11 22:25:30.411088 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.411053    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"]
Sep 11 22:25:30.415377 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.415353    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d09c96-7f59c6bf54-7tjx2"]
Sep 11 22:25:30.424196 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.424175    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"]
Sep 11 22:25:30.430859 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.430837    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d09c96-689bf87b69-h6jnw"]
Sep 11 22:25:30.856095 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.856063    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30854b6d-6fc1-4d3b-8e27-edef38c380a7" path="/var/lib/kubelet/pods/30854b6d-6fc1-4d3b-8e27-edef38c380a7/volumes"
Sep 11 22:25:30.856433 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.856420    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54933a8e-de7f-44a9-9c55-b21e3ece54a8" path="/var/lib/kubelet/pods/54933a8e-de7f-44a9-9c55-b21e3ece54a8/volumes"
Sep 11 22:25:30.856684 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:30.856671    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c50d293-982b-42a1-88ac-df758a1504f8" path="/var/lib/kubelet/pods/9c50d293-982b-42a1-88ac-df758a1504f8/volumes"
Sep 11 22:25:47.425769 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.425733    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"]
Sep 11 22:25:47.427906 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.427879    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-966527-29819415-2klrc"]
Sep 11 22:25:47.449971 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.449934    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"]
Sep 11 22:25:47.450269 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.450209    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9" podUID="33ec7970-dab7-4f5e-97d0-556fd6bcad14" containerName="payload-processing" containerID="cri-o://473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c" gracePeriod=30
Sep 11 22:25:47.458980 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.458933    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"]
Sep 11 22:25:47.459255 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.459193    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp" podUID="ff4770dc-eb19-4b83-854c-93a4603616b7" containerName="payload-pre-processing" containerID="cri-o://b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3" gracePeriod=30
Sep 11 22:25:47.735142 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.735111    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:25:47.738050 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.738030    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:25:47.894259 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.894225    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n62dr\" (UniqueName: \"kubernetes.io/projected/ff4770dc-eb19-4b83-854c-93a4603616b7-kube-api-access-n62dr\") pod \"ff4770dc-eb19-4b83-854c-93a4603616b7\" (UID: \"ff4770dc-eb19-4b83-854c-93a4603616b7\") "
Sep 11 22:25:47.894453 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.894292    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4770dc-eb19-4b83-854c-93a4603616b7-plugins-config-volume\") pod \"ff4770dc-eb19-4b83-854c-93a4603616b7\" (UID: \"ff4770dc-eb19-4b83-854c-93a4603616b7\") "
Sep 11 22:25:47.894453 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.894354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ec7970-dab7-4f5e-97d0-556fd6bcad14-plugins-config-volume\") pod \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\" (UID: \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\") "
Sep 11 22:25:47.894453 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.894400    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlcxd\" (UniqueName: \"kubernetes.io/projected/33ec7970-dab7-4f5e-97d0-556fd6bcad14-kube-api-access-hlcxd\") pod \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\" (UID: \"33ec7970-dab7-4f5e-97d0-556fd6bcad14\") "
Sep 11 22:25:47.894766 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.894734    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ec7970-dab7-4f5e-97d0-556fd6bcad14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33ec7970-dab7-4f5e-97d0-556fd6bcad14" (UID: "33ec7970-dab7-4f5e-97d0-556fd6bcad14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:47.894841 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.894768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4770dc-eb19-4b83-854c-93a4603616b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff4770dc-eb19-4b83-854c-93a4603616b7" (UID: "ff4770dc-eb19-4b83-854c-93a4603616b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:47.896567 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.896541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ec7970-dab7-4f5e-97d0-556fd6bcad14-kube-api-access-hlcxd" (OuterVolumeSpecName: "kube-api-access-hlcxd") pod "33ec7970-dab7-4f5e-97d0-556fd6bcad14" (UID: "33ec7970-dab7-4f5e-97d0-556fd6bcad14"). InnerVolumeSpecName "kube-api-access-hlcxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:47.896700 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.896581    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4770dc-eb19-4b83-854c-93a4603616b7-kube-api-access-n62dr" (OuterVolumeSpecName: "kube-api-access-n62dr") pod "ff4770dc-eb19-4b83-854c-93a4603616b7" (UID: "ff4770dc-eb19-4b83-854c-93a4603616b7"). InnerVolumeSpecName "kube-api-access-n62dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:47.995463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.995377    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ec7970-dab7-4f5e-97d0-556fd6bcad14-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:47.995463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.995406    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlcxd\" (UniqueName: \"kubernetes.io/projected/33ec7970-dab7-4f5e-97d0-556fd6bcad14-kube-api-access-hlcxd\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:47.995463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.995416    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62dr\" (UniqueName: \"kubernetes.io/projected/ff4770dc-eb19-4b83-854c-93a4603616b7-kube-api-access-n62dr\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:47.995463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:47.995425    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4770dc-eb19-4b83-854c-93a4603616b7-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:48.490533 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.490491    2578 generic.go:356] "Generic (PLEG): container finished" podID="ff4770dc-eb19-4b83-854c-93a4603616b7" containerID="b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3" exitCode=0
Sep 11 22:25:48.491295 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.490558    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"
Sep 11 22:25:48.491295 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.490591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp" event={"ID":"ff4770dc-eb19-4b83-854c-93a4603616b7","Type":"ContainerDied","Data":"b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3"}
Sep 11 22:25:48.491295 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.490722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp" event={"ID":"ff4770dc-eb19-4b83-854c-93a4603616b7","Type":"ContainerDied","Data":"4a5a6cf2f6f13f11d2ff0fb118b4f6aa5abcf72dac559cda0134fd04aee7eb67"}
Sep 11 22:25:48.491295 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.490761    2578 scope.go:117] "RemoveContainer" containerID="b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3"
Sep 11 22:25:48.492374 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.492351    2578 generic.go:356] "Generic (PLEG): container finished" podID="33ec7970-dab7-4f5e-97d0-556fd6bcad14" containerID="473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c" exitCode=0
Sep 11 22:25:48.492476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.492408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9" event={"ID":"33ec7970-dab7-4f5e-97d0-556fd6bcad14","Type":"ContainerDied","Data":"473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c"}
Sep 11 22:25:48.492476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.492418    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"
Sep 11 22:25:48.492586 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.492479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9" event={"ID":"33ec7970-dab7-4f5e-97d0-556fd6bcad14","Type":"ContainerDied","Data":"9314a1d7de78191d192172f691bbdd4c411c33b81ed3af33433b96acd70dbc7b"}
Sep 11 22:25:48.500728 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.500708    2578 scope.go:117] "RemoveContainer" containerID="b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3"
Sep 11 22:25:48.501161 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:25:48.501135    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3\": container with ID starting with b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3 not found: ID does not exist" containerID="b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3"
Sep 11 22:25:48.501256 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.501171    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3"} err="failed to get container status \"b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3\": rpc error: code = NotFound desc = could not find container \"b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3\": container with ID starting with b884e58cd8c18b9e6f12d621207fb58b7d06e03a20ab4ed43ae9e4aec1d52fb3 not found: ID does not exist"
Sep 11 22:25:48.501256 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.501191    2578 scope.go:117] "RemoveContainer" containerID="473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c"
Sep 11 22:25:48.509810 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.509794    2578 scope.go:117] "RemoveContainer" containerID="473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c"
Sep 11 22:25:48.510105 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:25:48.510086    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c\": container with ID starting with 473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c not found: ID does not exist" containerID="473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c"
Sep 11 22:25:48.510165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.510113    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c"} err="failed to get container status \"473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c\": rpc error: code = NotFound desc = could not find container \"473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c\": container with ID starting with 473e5f0fb56b54f6b151a805bc275b6fdfe0079c0a35ba2bee036797c50b1d5c not found: ID does not exist"
Sep 11 22:25:48.515380 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.515358    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"]
Sep 11 22:25:48.519738 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.519718    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-966527-8ccfd888b-wgzvp"]
Sep 11 22:25:48.529384 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.529361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"]
Sep 11 22:25:48.531287 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.531266    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-966527-57dfb5dcc9-tdhs9"]
Sep 11 22:25:48.858984 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.858901    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ec7970-dab7-4f5e-97d0-556fd6bcad14" path="/var/lib/kubelet/pods/33ec7970-dab7-4f5e-97d0-556fd6bcad14/volumes"
Sep 11 22:25:48.859195 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.859180    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58959601-b65d-4f5d-b067-96c6a5e293db" path="/var/lib/kubelet/pods/58959601-b65d-4f5d-b067-96c6a5e293db/volumes"
Sep 11 22:25:48.859470 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:48.859458    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff4770dc-eb19-4b83-854c-93a4603616b7" path="/var/lib/kubelet/pods/ff4770dc-eb19-4b83-854c-93a4603616b7/volumes"
Sep 11 22:25:54.339858 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:54.339822    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"]
Sep 11 22:25:54.340311 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:54.340082    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy" containerID="cri-o://9d933a6879bc4585a2bf62607315e776e629c0d0b812927fdcbd58198128833c" gracePeriod=30
Sep 11 22:25:59.561628 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.561586    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerID="9d933a6879bc4585a2bf62607315e776e629c0d0b812927fdcbd58198128833c" exitCode=0
Sep 11 22:25:59.562078 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.561665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" event={"ID":"9d63d2ef-3367-4a52-a706-8ba66a1c17c6","Type":"ContainerDied","Data":"9d933a6879bc4585a2bf62607315e776e629c0d0b812927fdcbd58198128833c"}
Sep 11 22:25:59.562078 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.561772    2578 scope.go:117] "RemoveContainer" containerID="a93ba515074d44d88fbc611d1ba88e5c885c7e672da904ac1c8207b1c79f36e7"
Sep 11 22:25:59.594289 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.594258    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:25:59.697263 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5wt4\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-kube-api-access-n5wt4\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697284    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-credential-socket\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697345    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-envoy\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697404    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-certs\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-token\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697463 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697452    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-data\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697756 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istiod-ca-cert\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697756 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-socket\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697756 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-podinfo\") pod \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\" (UID: \"9d63d2ef-3367-4a52-a706-8ba66a1c17c6\") "
Sep 11 22:25:59.697756 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697631    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:59.697970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697724    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:59.697970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697888    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:59.697970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697907    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:59.697970 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697932    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:59.698132 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.697969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:59.698168 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.698127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:59.700053 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.700019    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:59.700165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.700026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:25:59.700165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.700096    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:59.700165 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.700104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-kube-api-access-n5wt4" (OuterVolumeSpecName: "kube-api-access-n5wt4") pod "9d63d2ef-3367-4a52-a706-8ba66a1c17c6" (UID: "9d63d2ef-3367-4a52-a706-8ba66a1c17c6"). InnerVolumeSpecName "kube-api-access-n5wt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:59.798341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.798301    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:59.798341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.798335    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:59.798341 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.798344    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:59.798612 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.798355    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5wt4\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-kube-api-access-n5wt4\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:59.798612 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.798364    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:59.798612 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.798372    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:25:59.798612 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:25:59.798380    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d63d2ef-3367-4a52-a706-8ba66a1c17c6-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:26:00.568867 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:26:00.568839    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"
Sep 11 22:26:00.569421 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:26:00.568874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6" event={"ID":"9d63d2ef-3367-4a52-a706-8ba66a1c17c6","Type":"ContainerDied","Data":"a8ec017252001b7cc75c78140a28e03db004ea9463b188e6f7fd6a01937c3d82"}
Sep 11 22:26:00.569421 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:26:00.568939    2578 scope.go:117] "RemoveContainer" containerID="9d933a6879bc4585a2bf62607315e776e629c0d0b812927fdcbd58198128833c"
Sep 11 22:26:00.589859 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:26:00.589829    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"]
Sep 11 22:26:00.592745 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:26:00.592717    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-966527-openshift-default-6b766d4c9b-66hx6"]
Sep 11 22:26:00.856686 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:26:00.856595    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" path="/var/lib/kubelet/pods/9d63d2ef-3367-4a52-a706-8ba66a1c17c6/volumes"
Sep 11 22:26:19.376754 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:26:19.376718    2578 scope.go:117] "RemoveContainer" containerID="bf7b03d09528795b0a24069c56ada3801898e6db7b2c3c55f3040f870d2cea66"
Sep 11 22:26:19.385171 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:26:19.385151    2578 scope.go:117] "RemoveContainer" containerID="f6dce56c46719160ed8b9dd2527f1854a8459cdc6eeaa07ee23c22bb29f610b8"
Sep 11 22:30:00.134662 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134616    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5"]
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134944    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134955    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134962    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d83a84-c63a-4c6c-9c26-150321a68ff5" containerName="revoke-keys"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134968    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d83a84-c63a-4c6c-9c26-150321a68ff5" containerName="revoke-keys"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134976    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ec7970-dab7-4f5e-97d0-556fd6bcad14" containerName="payload-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134983    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ec7970-dab7-4f5e-97d0-556fd6bcad14" containerName="payload-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134992    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54933a8e-de7f-44a9-9c55-b21e3ece54a8" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.134997    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54933a8e-de7f-44a9-9c55-b21e3ece54a8" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135004    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff4770dc-eb19-4b83-854c-93a4603616b7" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135010    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff4770dc-eb19-4b83-854c-93a4603616b7" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135022    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135026    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135033    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30854b6d-6fc1-4d3b-8e27-edef38c380a7" containerName="payload-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135038    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30854b6d-6fc1-4d3b-8e27-edef38c380a7" containerName="payload-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135044    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a7eaed-ca97-48b5-8878-a33984e8e6e8" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135050    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a7eaed-ca97-48b5-8878-a33984e8e6e8" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135097    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d83a84-c63a-4c6c-9c26-150321a68ff5" containerName="revoke-keys"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135104    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135112    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30854b6d-6fc1-4d3b-8e27-edef38c380a7" containerName="payload-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135123    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="54933a8e-de7f-44a9-9c55-b21e3ece54a8" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135132    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d63d2ef-3367-4a52-a706-8ba66a1c17c6" containerName="istio-proxy"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135140    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff4770dc-eb19-4b83-854c-93a4603616b7" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135147    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a7eaed-ca97-48b5-8878-a33984e8e6e8" containerName="payload-pre-processing"
Sep 11 22:30:00.135332 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.135153    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ec7970-dab7-4f5e-97d0-556fd6bcad14" containerName="payload-processing"
Sep 11 22:30:00.137890 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.137874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5"
Sep 11 22:30:00.140690 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.140672    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6v6sw\""
Sep 11 22:30:00.148329 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.148295    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5"]
Sep 11 22:30:00.154378 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.154253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5"
Sep 11 22:30:00.428032 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.428004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5"]
Sep 11 22:30:00.430178 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:30:00.430152    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ff0eb8_f333_4eec_a476_cfa349ced6eb.slice/crio-87172310d0afeaae0f8101637f6ea68622541b7589509997eab7bbbaea25eac9 WatchSource:0}: Error finding container 87172310d0afeaae0f8101637f6ea68622541b7589509997eab7bbbaea25eac9: Status 404 returned error can't find the container with id 87172310d0afeaae0f8101637f6ea68622541b7589509997eab7bbbaea25eac9
Sep 11 22:30:00.801668 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.801555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5" event={"ID":"20ff0eb8-f333-4eec-a476-cfa349ced6eb","Type":"ContainerStarted","Data":"aecee9e0a7c0ce4ea670ca83acf677b2f08496d4e5fdff6a764906d8a73032d6"}
Sep 11 22:30:00.801668 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.801616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5" event={"ID":"20ff0eb8-f333-4eec-a476-cfa349ced6eb","Type":"ContainerStarted","Data":"87172310d0afeaae0f8101637f6ea68622541b7589509997eab7bbbaea25eac9"}
Sep 11 22:30:00.817370 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:00.817323    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5" podStartSLOduration=0.817307894 podStartE2EDuration="817.307894ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:30:00.815757217 +0000 UTC m=+2262.500703930" watchObservedRunningTime="2026-09-11 22:30:00.817307894 +0000 UTC m=+2262.502254609"
Sep 11 22:30:01.377040 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.377006    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"]
Sep 11 22:30:01.380711 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.380689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.382737 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.382714    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f0eecc64-kserve-self-signed-certs\""
Sep 11 22:30:01.389450 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.389425    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"]
Sep 11 22:30:01.501764 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.501725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-dshm\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.502009 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.501784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tls-certs\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.502009 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.501841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-model-cache\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.502009 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.501873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tmp-dir\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.502009 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.501959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-home\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.502242 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.502018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmmk\" (UniqueName: \"kubernetes.io/projected/14331cb3-25e9-466a-bc74-cd4c7aa11a59-kube-api-access-qvmmk\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603057 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.602971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tls-certs\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603057 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.603014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-model-cache\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603057 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.603035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tmp-dir\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603057 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.603059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-home\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603380 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.603092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmmk\" (UniqueName: \"kubernetes.io/projected/14331cb3-25e9-466a-bc74-cd4c7aa11a59-kube-api-access-qvmmk\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603380 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.603135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-dshm\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603566 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.603534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-model-cache\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603657 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.603588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tmp-dir\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.603657 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.603582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-home\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.605355 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.605331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-dshm\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.605659 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.605620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tls-certs\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.621537 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.621512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmmk\" (UniqueName: \"kubernetes.io/projected/14331cb3-25e9-466a-bc74-cd4c7aa11a59-kube-api-access-qvmmk\") pod \"e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") " pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.698679 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.698629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:01.808211 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.808173    2578 generic.go:356] "Generic (PLEG): container finished" podID="20ff0eb8-f333-4eec-a476-cfa349ced6eb" containerID="aecee9e0a7c0ce4ea670ca83acf677b2f08496d4e5fdff6a764906d8a73032d6" exitCode=0
Sep 11 22:30:01.808387 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.808250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5" event={"ID":"20ff0eb8-f333-4eec-a476-cfa349ced6eb","Type":"ContainerDied","Data":"aecee9e0a7c0ce4ea670ca83acf677b2f08496d4e5fdff6a764906d8a73032d6"}
Sep 11 22:30:01.970334 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:01.970305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"]
Sep 11 22:30:01.972205 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:30:01.972175    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14331cb3_25e9_466a_bc74_cd4c7aa11a59.slice/crio-b484819f3fd94033698e00a302c0dfe50d8993b09742144a61b96f82e0721275 WatchSource:0}: Error finding container b484819f3fd94033698e00a302c0dfe50d8993b09742144a61b96f82e0721275: Status 404 returned error can't find the container with id b484819f3fd94033698e00a302c0dfe50d8993b09742144a61b96f82e0721275
Sep 11 22:30:02.814675 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:02.814623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75" event={"ID":"14331cb3-25e9-466a-bc74-cd4c7aa11a59","Type":"ContainerStarted","Data":"22ab05811b738773636f5cffa16b5d9f50be1737e80b8c48d5b0e2a40f99b4ee"}
Sep 11 22:30:02.814675 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:02.814682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75" event={"ID":"14331cb3-25e9-466a-bc74-cd4c7aa11a59","Type":"ContainerStarted","Data":"b484819f3fd94033698e00a302c0dfe50d8993b09742144a61b96f82e0721275"}
Sep 11 22:30:02.832664 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:02.832604    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75" podStartSLOduration=1.8325866560000001 podStartE2EDuration="1.832586656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:30:02.830178504 +0000 UTC m=+2264.515125217" watchObservedRunningTime="2026-09-11 22:30:02.832586656 +0000 UTC m=+2264.517533401"
Sep 11 22:30:02.940557 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:02.940529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5"
Sep 11 22:30:03.820673 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:03.820615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5" event={"ID":"20ff0eb8-f333-4eec-a476-cfa349ced6eb","Type":"ContainerDied","Data":"87172310d0afeaae0f8101637f6ea68622541b7589509997eab7bbbaea25eac9"}
Sep 11 22:30:03.820673 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:03.820655    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819430-h7jc5"
Sep 11 22:30:03.821094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:03.820686    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87172310d0afeaae0f8101637f6ea68622541b7589509997eab7bbbaea25eac9"
Sep 11 22:30:10.378832 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.378803    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"]
Sep 11 22:30:10.379260 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.379137    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ff0eb8-f333-4eec-a476-cfa349ced6eb" containerName="cleanup"
Sep 11 22:30:10.379260 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.379149    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ff0eb8-f333-4eec-a476-cfa349ced6eb" containerName="cleanup"
Sep 11 22:30:10.379260 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.379210    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ff0eb8-f333-4eec-a476-cfa349ced6eb" containerName="cleanup"
Sep 11 22:30:10.382290 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.382268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.384365 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.384344    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f0eecc64-kserve-self-signed-certs\""
Sep 11 22:30:10.390710 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.390687    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"]
Sep 11 22:30:10.476523 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.476477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-tmp-dir\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.476523 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.476524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npcw\" (UniqueName: \"kubernetes.io/projected/41366b84-c01f-494c-b920-b401873d0b76-kube-api-access-7npcw\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.476782 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.476580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-home\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.476782 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.476656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41366b84-c01f-494c-b920-b401873d0b76-tls-certs\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.476782 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.476691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-dshm\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.476887 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.476797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-model-cache\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.577744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.577704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41366b84-c01f-494c-b920-b401873d0b76-tls-certs\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.577953 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.577759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-dshm\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.577953 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.577815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-model-cache\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.577953 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.577836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-tmp-dir\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.577953 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.577864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npcw\" (UniqueName: \"kubernetes.io/projected/41366b84-c01f-494c-b920-b401873d0b76-kube-api-access-7npcw\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.578182 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.577952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-home\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.578577 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.578434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-model-cache\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.578723 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.578458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-tmp-dir\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.578723 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.578482    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-home\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.580199 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.580164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-dshm\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.580487 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.580464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41366b84-c01f-494c-b920-b401873d0b76-tls-certs\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.585720 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.585695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npcw\" (UniqueName: \"kubernetes.io/projected/41366b84-c01f-494c-b920-b401873d0b76-kube-api-access-7npcw\") pod \"e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") " pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.691948 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.691911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:10.968753 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:10.968728    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"]
Sep 11 22:30:10.971087 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:30:10.971048    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41366b84_c01f_494c_b920_b401873d0b76.slice/crio-096e81ac86d4bf0f4456a451b5aa0dae7f0a0b2d684c1b7a7c127ae32ba3daa7 WatchSource:0}: Error finding container 096e81ac86d4bf0f4456a451b5aa0dae7f0a0b2d684c1b7a7c127ae32ba3daa7: Status 404 returned error can't find the container with id 096e81ac86d4bf0f4456a451b5aa0dae7f0a0b2d684c1b7a7c127ae32ba3daa7
Sep 11 22:30:11.699781 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:11.699742    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:11.699781 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:11.699792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:11.712388 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:11.712359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:11.865094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:11.865055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd" event={"ID":"41366b84-c01f-494c-b920-b401873d0b76","Type":"ContainerStarted","Data":"9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c"}
Sep 11 22:30:11.865094 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:11.865103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd" event={"ID":"41366b84-c01f-494c-b920-b401873d0b76","Type":"ContainerStarted","Data":"096e81ac86d4bf0f4456a451b5aa0dae7f0a0b2d684c1b7a7c127ae32ba3daa7"}
Sep 11 22:30:11.876149 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:11.876122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:30:11.905492 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:11.905436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd" podStartSLOduration=1.905421736 podStartE2EDuration="1.905421736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:30:11.902270036 +0000 UTC m=+2273.587216748" watchObservedRunningTime="2026-09-11 22:30:11.905421736 +0000 UTC m=+2273.590368497"
Sep 11 22:30:19.451157 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:19.451120    2578 scope.go:117] "RemoveContainer" containerID="f66134e7fa3b9bfd59299a675b5878ab786b6a0a02d7bad25e82a756c84ff6c1"
Sep 11 22:30:19.459887 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:19.459863    2578 scope.go:117] "RemoveContainer" containerID="c0bcf3c82342646027c7ab4576107762d4cc19c9a865483dfae2afdd31088f13"
Sep 11 22:30:20.692927 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:20.692892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:20.693291 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:20.692944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:20.705496 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:20.705465    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:20.921796 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:20.921765    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:30:48.897075 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:48.897039    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"]
Sep 11 22:30:49.691838 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:30:49.691804    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"]
Sep 11 22:31:03.918425 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:03.918382    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75" podUID="14331cb3-25e9-466a-bc74-cd4c7aa11a59" containerName="main" containerID="cri-o://22ab05811b738773636f5cffa16b5d9f50be1737e80b8c48d5b0e2a40f99b4ee" gracePeriod=45
Sep 11 22:31:04.142527 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.142497    2578 generic.go:356] "Generic (PLEG): container finished" podID="14331cb3-25e9-466a-bc74-cd4c7aa11a59" containerID="22ab05811b738773636f5cffa16b5d9f50be1737e80b8c48d5b0e2a40f99b4ee" exitCode=0
Sep 11 22:31:04.142677 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.142565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75" event={"ID":"14331cb3-25e9-466a-bc74-cd4c7aa11a59","Type":"ContainerDied","Data":"22ab05811b738773636f5cffa16b5d9f50be1737e80b8c48d5b0e2a40f99b4ee"}
Sep 11 22:31:04.159841 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.159814    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:31:04.264766 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.264672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tmp-dir\") pod \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") "
Sep 11 22:31:04.264766 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.264715    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tls-certs\") pod \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") "
Sep 11 22:31:04.265007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.264786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-model-cache\") pod \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") "
Sep 11 22:31:04.265007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.264803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-home\") pod \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") "
Sep 11 22:31:04.265007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.264823    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvmmk\" (UniqueName: \"kubernetes.io/projected/14331cb3-25e9-466a-bc74-cd4c7aa11a59-kube-api-access-qvmmk\") pod \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") "
Sep 11 22:31:04.265007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.264911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-dshm\") pod \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\" (UID: \"14331cb3-25e9-466a-bc74-cd4c7aa11a59\") "
Sep 11 22:31:04.265007 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.264973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14331cb3-25e9-466a-bc74-cd4c7aa11a59" (UID: "14331cb3-25e9-466a-bc74-cd4c7aa11a59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:31:04.265240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.265084    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-model-cache" (OuterVolumeSpecName: "model-cache") pod "14331cb3-25e9-466a-bc74-cd4c7aa11a59" (UID: "14331cb3-25e9-466a-bc74-cd4c7aa11a59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:31:04.265240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.265100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-home" (OuterVolumeSpecName: "home") pod "14331cb3-25e9-466a-bc74-cd4c7aa11a59" (UID: "14331cb3-25e9-466a-bc74-cd4c7aa11a59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:31:04.265313 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.265246    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:04.265313 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.265258    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:04.265313 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.265268    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:04.267138 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.267106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-dshm" (OuterVolumeSpecName: "dshm") pod "14331cb3-25e9-466a-bc74-cd4c7aa11a59" (UID: "14331cb3-25e9-466a-bc74-cd4c7aa11a59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:31:04.267257 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.267170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14331cb3-25e9-466a-bc74-cd4c7aa11a59-kube-api-access-qvmmk" (OuterVolumeSpecName: "kube-api-access-qvmmk") pod "14331cb3-25e9-466a-bc74-cd4c7aa11a59" (UID: "14331cb3-25e9-466a-bc74-cd4c7aa11a59"). InnerVolumeSpecName "kube-api-access-qvmmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:31:04.267257 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.267179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14331cb3-25e9-466a-bc74-cd4c7aa11a59" (UID: "14331cb3-25e9-466a-bc74-cd4c7aa11a59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:31:04.366516 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.366479    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14331cb3-25e9-466a-bc74-cd4c7aa11a59-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:04.366516 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.366506    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmmk\" (UniqueName: \"kubernetes.io/projected/14331cb3-25e9-466a-bc74-cd4c7aa11a59-kube-api-access-qvmmk\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:04.366516 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.366517    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14331cb3-25e9-466a-bc74-cd4c7aa11a59-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:04.711287 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.711219    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd" podUID="41366b84-c01f-494c-b920-b401873d0b76" containerName="main" containerID="cri-o://9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c" gracePeriod=45
Sep 11 22:31:04.953257 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.953231    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:31:04.971286 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971218    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41366b84-c01f-494c-b920-b401873d0b76-tls-certs\") pod \"41366b84-c01f-494c-b920-b401873d0b76\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") "
Sep 11 22:31:04.971286 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-dshm\") pod \"41366b84-c01f-494c-b920-b401873d0b76\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") "
Sep 11 22:31:04.971476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971306    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7npcw\" (UniqueName: \"kubernetes.io/projected/41366b84-c01f-494c-b920-b401873d0b76-kube-api-access-7npcw\") pod \"41366b84-c01f-494c-b920-b401873d0b76\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") "
Sep 11 22:31:04.971476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-model-cache\") pod \"41366b84-c01f-494c-b920-b401873d0b76\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") "
Sep 11 22:31:04.971476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-home\") pod \"41366b84-c01f-494c-b920-b401873d0b76\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") "
Sep 11 22:31:04.971476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-tmp-dir\") pod \"41366b84-c01f-494c-b920-b401873d0b76\" (UID: \"41366b84-c01f-494c-b920-b401873d0b76\") "
Sep 11 22:31:04.971850 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-home" (OuterVolumeSpecName: "home") pod "41366b84-c01f-494c-b920-b401873d0b76" (UID: "41366b84-c01f-494c-b920-b401873d0b76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:31:04.971956 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971914    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41366b84-c01f-494c-b920-b401873d0b76" (UID: "41366b84-c01f-494c-b920-b401873d0b76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:31:04.971956 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.971898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-model-cache" (OuterVolumeSpecName: "model-cache") pod "41366b84-c01f-494c-b920-b401873d0b76" (UID: "41366b84-c01f-494c-b920-b401873d0b76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:31:04.973837 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.973800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-dshm" (OuterVolumeSpecName: "dshm") pod "41366b84-c01f-494c-b920-b401873d0b76" (UID: "41366b84-c01f-494c-b920-b401873d0b76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:31:04.973963 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.973939    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41366b84-c01f-494c-b920-b401873d0b76-kube-api-access-7npcw" (OuterVolumeSpecName: "kube-api-access-7npcw") pod "41366b84-c01f-494c-b920-b401873d0b76" (UID: "41366b84-c01f-494c-b920-b401873d0b76"). InnerVolumeSpecName "kube-api-access-7npcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:31:04.973963 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:04.973951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41366b84-c01f-494c-b920-b401873d0b76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41366b84-c01f-494c-b920-b401873d0b76" (UID: "41366b84-c01f-494c-b920-b401873d0b76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:31:05.072479 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.072440    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7npcw\" (UniqueName: \"kubernetes.io/projected/41366b84-c01f-494c-b920-b401873d0b76-kube-api-access-7npcw\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:05.072479 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.072474    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:05.072479 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.072483    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:05.072744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.072491    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:05.072744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.072500    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41366b84-c01f-494c-b920-b401873d0b76-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:05.072744 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.072507    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41366b84-c01f-494c-b920-b401873d0b76-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 11 22:31:05.154587 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.154556    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"
Sep 11 22:31:05.154756 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.154555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75" event={"ID":"14331cb3-25e9-466a-bc74-cd4c7aa11a59","Type":"ContainerDied","Data":"b484819f3fd94033698e00a302c0dfe50d8993b09742144a61b96f82e0721275"}
Sep 11 22:31:05.154756 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.154689    2578 scope.go:117] "RemoveContainer" containerID="22ab05811b738773636f5cffa16b5d9f50be1737e80b8c48d5b0e2a40f99b4ee"
Sep 11 22:31:05.156082 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.156057    2578 generic.go:356] "Generic (PLEG): container finished" podID="41366b84-c01f-494c-b920-b401873d0b76" containerID="9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c" exitCode=0
Sep 11 22:31:05.156180 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.156122    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"
Sep 11 22:31:05.156237 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.156122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd" event={"ID":"41366b84-c01f-494c-b920-b401873d0b76","Type":"ContainerDied","Data":"9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c"}
Sep 11 22:31:05.156286 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.156237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd" event={"ID":"41366b84-c01f-494c-b920-b401873d0b76","Type":"ContainerDied","Data":"096e81ac86d4bf0f4456a451b5aa0dae7f0a0b2d684c1b7a7c127ae32ba3daa7"}
Sep 11 22:31:05.164473 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.164451    2578 scope.go:117] "RemoveContainer" containerID="9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c"
Sep 11 22:31:05.172387 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.172365    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"]
Sep 11 22:31:05.172651 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.172620    2578 scope.go:117] "RemoveContainer" containerID="9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c"
Sep 11 22:31:05.172926 ip-10-0-141-45 kubenswrapper[2578]: E0911 22:31:05.172906    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c\": container with ID starting with 9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c not found: ID does not exist" containerID="9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c"
Sep 11 22:31:05.172926 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.172937    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c"} err="failed to get container status \"9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c\": rpc error: code = NotFound desc = could not find container \"9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c\": container with ID starting with 9c2c421121dc920c58ce9359649144366f7ed4d2f838081033cb50f1f936ef0c not found: ID does not exist"
Sep 11 22:31:05.175935 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.175914    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f0eecc64-kserve-86b84b9b79-5cs75"]
Sep 11 22:31:05.187016 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.186990    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"]
Sep 11 22:31:05.190016 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:05.189992    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f0eecc64-kserve-65c7449f87-g7jgd"]
Sep 11 22:31:06.856214 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:06.856179    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14331cb3-25e9-466a-bc74-cd4c7aa11a59" path="/var/lib/kubelet/pods/14331cb3-25e9-466a-bc74-cd4c7aa11a59/volumes"
Sep 11 22:31:06.856597 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:06.856580    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41366b84-c01f-494c-b920-b401873d0b76" path="/var/lib/kubelet/pods/41366b84-c01f-494c-b920-b401873d0b76/volumes"
Sep 11 22:31:19.483239 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:19.483159    2578 scope.go:117] "RemoveContainer" containerID="15917603273ef4409a0815f43dc680990a1f79ab6c01d7b850e7ea12d176020c"
Sep 11 22:31:44.032085 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:44.032053    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-jpctb_0ad12098-b953-49ec-8fbe-9907801c0c62/manager/0.log"
Sep 11 22:31:44.038955 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:44.038932    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-jpctb_0ad12098-b953-49ec-8fbe-9907801c0c62/copy-manifests/0.log"
Sep 11 22:31:44.842239 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:44.842209    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819430-h7jc5_20ff0eb8-f333-4eec-a476-cfa349ced6eb/cleanup/0.log"
Sep 11 22:31:44.952295 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:44.952258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-sp24z_0a43fa42-09c2-457b-b1ac-ce4860f754a5/postgres/0.log"
Sep 11 22:31:46.847111 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:46.847078    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx_93eece28-e770-46f9-890e-4b59e58cbacc/istio-proxy/0.log"
Sep 11 22:31:47.091355 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:47.091312    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b4b4bb7-8g9g5_524ca142-8f37-48be-b646-5ac01dc75c1c/kube-auth-proxy/0.log"
Sep 11 22:31:48.034511 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:48.034473    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-8tlm6_ed0d7638-f84e-4fa9-a0ea-6ba35e7292e2/main/0.log"
Sep 11 22:31:48.390939 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:48.390853    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-d7lz5_faf1376e-e355-4d21-b60a-31ea3b4ba2dd/main/0.log"
Sep 11 22:31:48.733986 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:48.733958    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pxnb2_40625582-367e-4eb8-8df4-2622edae4c10/main/0.log"
Sep 11 22:31:54.790993 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.790954    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbnb9/must-gather-t2glf"]
Sep 11 22:31:54.791476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.791273    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41366b84-c01f-494c-b920-b401873d0b76" containerName="main"
Sep 11 22:31:54.791476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.791284    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41366b84-c01f-494c-b920-b401873d0b76" containerName="main"
Sep 11 22:31:54.791476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.791308    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14331cb3-25e9-466a-bc74-cd4c7aa11a59" containerName="main"
Sep 11 22:31:54.791476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.791314    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14331cb3-25e9-466a-bc74-cd4c7aa11a59" containerName="main"
Sep 11 22:31:54.791476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.791379    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41366b84-c01f-494c-b920-b401873d0b76" containerName="main"
Sep 11 22:31:54.791476 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.791389    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="14331cb3-25e9-466a-bc74-cd4c7aa11a59" containerName="main"
Sep 11 22:31:54.794921 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.794898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbnb9/must-gather-t2glf"
Sep 11 22:31:54.797274 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.797249    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbnb9\"/\"kube-root-ca.crt\""
Sep 11 22:31:54.797862 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.797838    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbnb9\"/\"openshift-service-ca.crt\""
Sep 11 22:31:54.797992 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.797859    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vbnb9\"/\"default-dockercfg-pz8hq\""
Sep 11 22:31:54.800996 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.800975    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbnb9/must-gather-t2glf"]
Sep 11 22:31:54.801939 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.801919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/841db1a0-7ef8-4363-b799-ebac9fbb27ca-must-gather-output\") pod \"must-gather-t2glf\" (UID: \"841db1a0-7ef8-4363-b799-ebac9fbb27ca\") " pod="openshift-must-gather-vbnb9/must-gather-t2glf"
Sep 11 22:31:54.802034 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.801971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqxdm\" (UniqueName: \"kubernetes.io/projected/841db1a0-7ef8-4363-b799-ebac9fbb27ca-kube-api-access-xqxdm\") pod \"must-gather-t2glf\" (UID: \"841db1a0-7ef8-4363-b799-ebac9fbb27ca\") " pod="openshift-must-gather-vbnb9/must-gather-t2glf"
Sep 11 22:31:54.902579 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.902534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqxdm\" (UniqueName: \"kubernetes.io/projected/841db1a0-7ef8-4363-b799-ebac9fbb27ca-kube-api-access-xqxdm\") pod \"must-gather-t2glf\" (UID: \"841db1a0-7ef8-4363-b799-ebac9fbb27ca\") " pod="openshift-must-gather-vbnb9/must-gather-t2glf"
Sep 11 22:31:54.902817 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.902626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/841db1a0-7ef8-4363-b799-ebac9fbb27ca-must-gather-output\") pod \"must-gather-t2glf\" (UID: \"841db1a0-7ef8-4363-b799-ebac9fbb27ca\") " pod="openshift-must-gather-vbnb9/must-gather-t2glf"
Sep 11 22:31:54.903030 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.903009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/841db1a0-7ef8-4363-b799-ebac9fbb27ca-must-gather-output\") pod \"must-gather-t2glf\" (UID: \"841db1a0-7ef8-4363-b799-ebac9fbb27ca\") " pod="openshift-must-gather-vbnb9/must-gather-t2glf"
Sep 11 22:31:54.910371 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:54.910351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqxdm\" (UniqueName: \"kubernetes.io/projected/841db1a0-7ef8-4363-b799-ebac9fbb27ca-kube-api-access-xqxdm\") pod \"must-gather-t2glf\" (UID: \"841db1a0-7ef8-4363-b799-ebac9fbb27ca\") " pod="openshift-must-gather-vbnb9/must-gather-t2glf"
Sep 11 22:31:55.105431 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:55.105328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbnb9/must-gather-t2glf"
Sep 11 22:31:55.387098 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:55.387070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbnb9/must-gather-t2glf"]
Sep 11 22:31:55.389125 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:31:55.389094    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod841db1a0_7ef8_4363_b799_ebac9fbb27ca.slice/crio-021ec13ef20ee35e4704b10519e427d7b8ddd112f9c19d6487e11717ac5d2848 WatchSource:0}: Error finding container 021ec13ef20ee35e4704b10519e427d7b8ddd112f9c19d6487e11717ac5d2848: Status 404 returned error can't find the container with id 021ec13ef20ee35e4704b10519e427d7b8ddd112f9c19d6487e11717ac5d2848
Sep 11 22:31:55.391130 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:55.391108    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:31:55.417450 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:55.417414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/must-gather-t2glf" event={"ID":"841db1a0-7ef8-4363-b799-ebac9fbb27ca","Type":"ContainerStarted","Data":"021ec13ef20ee35e4704b10519e427d7b8ddd112f9c19d6487e11717ac5d2848"}
Sep 11 22:31:56.427082 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:56.427042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/must-gather-t2glf" event={"ID":"841db1a0-7ef8-4363-b799-ebac9fbb27ca","Type":"ContainerStarted","Data":"994086cbd64b485096c5f76872e0ef379d9ce1c810f5a0ca0d0a58fe63be8506"}
Sep 11 22:31:56.427679 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:56.427621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/must-gather-t2glf" event={"ID":"841db1a0-7ef8-4363-b799-ebac9fbb27ca","Type":"ContainerStarted","Data":"5481bbb701d73eb320030a4494000e562b39eef285674b7acb2743e154d0b8f5"}
Sep 11 22:31:56.443999 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:56.443948    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbnb9/must-gather-t2glf" podStartSLOduration=1.803976266 podStartE2EDuration="2.443933126s" totalImagesPullingTime="639.95686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:31:56.44235989 +0000 UTC m=+2378.127306603" watchObservedRunningTime="2026-09-11 22:31:56.443933126 +0000 UTC m=+2378.128879839"
Sep 11 22:31:57.662160 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:57.662120    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cq4fq_e8cbae59-1fe9-4805-b823-dc872f45dabc/global-pull-secret-syncer/0.log"
Sep 11 22:31:57.841286 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:57.841260    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x7k89_3aca9477-2265-43f3-b1e1-ba961826dd02/konnectivity-agent/0.log"
Sep 11 22:31:57.910296 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:31:57.910267    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-45.ec2.internal_9fec26c83d95126dedd1e7e37e73c673/haproxy/0.log"
Sep 11 22:32:04.433454 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.433424    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65b49caf-7abf-4067-a6cc-366ae9016753/alertmanager/0.log"
Sep 11 22:32:04.457404 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.457315    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65b49caf-7abf-4067-a6cc-366ae9016753/config-reloader/0.log"
Sep 11 22:32:04.485751 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.485596    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65b49caf-7abf-4067-a6cc-366ae9016753/kube-rbac-proxy-web/0.log"
Sep 11 22:32:04.507586 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.507560    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65b49caf-7abf-4067-a6cc-366ae9016753/kube-rbac-proxy/0.log"
Sep 11 22:32:04.533872 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.533846    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65b49caf-7abf-4067-a6cc-366ae9016753/kube-rbac-proxy-metric/0.log"
Sep 11 22:32:04.557112 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.557085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65b49caf-7abf-4067-a6cc-366ae9016753/prom-label-proxy/0.log"
Sep 11 22:32:04.588001 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.587975    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65b49caf-7abf-4067-a6cc-366ae9016753/init-config-reloader/0.log"
Sep 11 22:32:04.697711 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.696134    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rf89z_61e0d08f-9078-4087-bf40-fc0457d37a8d/kube-state-metrics/0.log"
Sep 11 22:32:04.742677 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.742623    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rf89z_61e0d08f-9078-4087-bf40-fc0457d37a8d/kube-rbac-proxy-main/0.log"
Sep 11 22:32:04.776797 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:04.776764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rf89z_61e0d08f-9078-4087-bf40-fc0457d37a8d/kube-rbac-proxy-self/0.log"
Sep 11 22:32:05.071002 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:05.070918    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2mw7_b88eb3f2-f059-4de3-8403-963ca05463c5/node-exporter/0.log"
Sep 11 22:32:05.095601 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:05.095570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2mw7_b88eb3f2-f059-4de3-8403-963ca05463c5/kube-rbac-proxy/0.log"
Sep 11 22:32:05.122362 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:05.122325    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2mw7_b88eb3f2-f059-4de3-8403-963ca05463c5/init-textfile/0.log"
Sep 11 22:32:05.502306 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:05.502273    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-q95dx_a06027be-dccb-43e0-a8aa-35971bdca8b9/prometheus-operator-admission-webhook/0.log"
Sep 11 22:32:07.765548 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.765508    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"]
Sep 11 22:32:07.771884 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.771852    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.774906 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.774878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"]
Sep 11 22:32:07.837450 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.837136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-sys\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.837450 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.837208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-lib-modules\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.837450 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.837240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-podres\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.837450 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.837267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-proc\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.837450 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.837324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8w48\" (UniqueName: \"kubernetes.io/projected/f7800bce-c22c-4692-a380-6af3ce82246b-kube-api-access-b8w48\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.938283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-lib-modules\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.938333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-podres\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.938363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-proc\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.938419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8w48\" (UniqueName: \"kubernetes.io/projected/f7800bce-c22c-4692-a380-6af3ce82246b-kube-api-access-b8w48\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.938594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-sys\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.938720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-sys\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.939123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-lib-modules\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.939197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-podres\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.939987 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.939244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7800bce-c22c-4692-a380-6af3ce82246b-proc\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:07.948467 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:07.948412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8w48\" (UniqueName: \"kubernetes.io/projected/f7800bce-c22c-4692-a380-6af3ce82246b-kube-api-access-b8w48\") pod \"perf-node-gather-daemonset-9dcj7\" (UID: \"f7800bce-c22c-4692-a380-6af3ce82246b\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:08.092295 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:08.092197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:08.386985 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:08.386930    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"]
Sep 11 22:32:08.389342 ip-10-0-141-45 kubenswrapper[2578]: W0911 22:32:08.389299    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf7800bce_c22c_4692_a380_6af3ce82246b.slice/crio-cd78b2bb8408144c41aa21236930f6efdc8038a4c18943cafcf64461c3acf8d7 WatchSource:0}: Error finding container cd78b2bb8408144c41aa21236930f6efdc8038a4c18943cafcf64461c3acf8d7: Status 404 returned error can't find the container with id cd78b2bb8408144c41aa21236930f6efdc8038a4c18943cafcf64461c3acf8d7
Sep 11 22:32:08.523240 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:08.523193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7" event={"ID":"f7800bce-c22c-4692-a380-6af3ce82246b","Type":"ContainerStarted","Data":"cd78b2bb8408144c41aa21236930f6efdc8038a4c18943cafcf64461c3acf8d7"}
Sep 11 22:32:09.531788 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:09.531748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7" event={"ID":"f7800bce-c22c-4692-a380-6af3ce82246b","Type":"ContainerStarted","Data":"d12370bbb37591841e3692a9801ef7c4cb7a1be29014b32ffa86ee13c5ddf031"}
Sep 11 22:32:09.532287 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:09.531892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:09.547870 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:09.547818    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7" podStartSLOduration=2.547802062 podStartE2EDuration="2.547802062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 22:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:32:09.545823719 +0000 UTC m=+2391.230770433" watchObservedRunningTime="2026-09-11 22:32:09.547802062 +0000 UTC m=+2391.232748776"
Sep 11 22:32:10.238667 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:10.238624    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mjmp7_9747109c-ab47-4d2d-8297-9b8b55d19d35/dns/0.log"
Sep 11 22:32:10.260243 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:10.260201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mjmp7_9747109c-ab47-4d2d-8297-9b8b55d19d35/kube-rbac-proxy/0.log"
Sep 11 22:32:10.403157 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:10.403124    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dj8qt_f4acaee8-78a0-4c04-bbab-ae5de79759a0/dns-node-resolver/0.log"
Sep 11 22:32:11.054381 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:11.054341    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h8vxg_826cf751-e2f9-4840-b136-6ca3241d2b29/node-ca/0.log"
Sep 11 22:32:12.085938 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:12.085875    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx_93eece28-e770-46f9-890e-4b59e58cbacc/istio-proxy/0.log"
Sep 11 22:32:12.529552 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:12.529500    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b4b4bb7-8g9g5_524ca142-8f37-48be-b646-5ac01dc75c1c/kube-auth-proxy/0.log"
Sep 11 22:32:13.606613 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:13.606570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tnsj_6e0f6fc4-3214-42ba-9777-62e327c1ded2/serve-healthcheck-canary/0.log"
Sep 11 22:32:14.148010 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:14.147978    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6l6gw_6c0d5d5a-e598-444e-b63a-208bafb0aaa5/kube-rbac-proxy/0.log"
Sep 11 22:32:14.172006 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:14.171981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6l6gw_6c0d5d5a-e598-444e-b63a-208bafb0aaa5/exporter/0.log"
Sep 11 22:32:14.195011 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:14.194980    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6l6gw_6c0d5d5a-e598-444e-b63a-208bafb0aaa5/extractor/0.log"
Sep 11 22:32:15.550689 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:15.550660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-9dcj7"
Sep 11 22:32:17.387854 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:17.387790    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-jpctb_0ad12098-b953-49ec-8fbe-9907801c0c62/manager/0.log"
Sep 11 22:32:17.413161 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:17.413134    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-jpctb_0ad12098-b953-49ec-8fbe-9907801c0c62/copy-manifests/0.log"
Sep 11 22:32:18.655660 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:18.655597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx_93eece28-e770-46f9-890e-4b59e58cbacc/istio-proxy/0.log"
Sep 11 22:32:18.942890 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:18.942797    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57494c65d5-t5768_6b0db6eb-463f-4c90-90b6-00ce94afd71b/manager/0.log"
Sep 11 22:32:19.011352 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:19.011320    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-827v4_6479942c-c6af-4cb5-8ff9-39c867d76981/openshift-lws-operator/0.log"
Sep 11 22:32:19.093259 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:19.092993    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b4b4bb7-8g9g5_524ca142-8f37-48be-b646-5ac01dc75c1c/kube-auth-proxy/0.log"
Sep 11 22:32:20.710099 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:20.710067    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-97vdx_93eece28-e770-46f9-890e-4b59e58cbacc/istio-proxy/0.log"
Sep 11 22:32:21.191084 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:21.191049    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b4b4bb7-8g9g5_524ca142-8f37-48be-b646-5ac01dc75c1c/kube-auth-proxy/0.log"
Sep 11 22:32:24.192075 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:24.192042    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-lz758_00997be9-004e-4c41-b42c-82bb1783d84c/migrator/0.log"
Sep 11 22:32:24.217072 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:24.217030    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-lz758_00997be9-004e-4c41-b42c-82bb1783d84c/graceful-termination/0.log"
Sep 11 22:32:25.746097 ip-10-0-141-45 kubenswrapper[2578]: I0911 22:32:25.746062    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5ncm8_9486e46e-ec91-4007-b19d-eed2ac5784bf/kube-multus-additional-cni-plugins/0.log"