Sep 16 00:43:34.183809 ip-10-0-131-89 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 00:43:34.554347 ip-10-0-131-89 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:43:34.554839 ip-10-0-131-89 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 00:43:34.554839 ip-10-0-131-89 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:43:34.554839 ip-10-0-131-89 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 00:43:34.554839 ip-10-0-131-89 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:43:34.554839 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.554437    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 00:43:34.558751 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558737    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:34.558751 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558752    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558757    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558761    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558764    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558767    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558770    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558773    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558776    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558779    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558782    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558786    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558788    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558791    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558794    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558796    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558799    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558801    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558804    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558806    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558809    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:34.558821 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558811    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558814    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558816    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558819    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558821    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558824    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558827    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558830    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558834    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558839    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558842    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558845    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558849    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558852    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558855    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558858    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558861    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558863    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558866    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558869    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:34.559333 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558871    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558874    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558876    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558879    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558882    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558884    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558887    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558889    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558892    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558894    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558897    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558899    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558902    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558905    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558908    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558913    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558917    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558919    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558922    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:34.559832 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558925    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558928    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558931    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558933    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558936    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558939    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558941    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558944    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558947    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558949    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558952    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558954    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558958    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558962    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558965    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558967    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558969    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558973    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558975    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:34.560295 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558978    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558980    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558983    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558985    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558988    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558990    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558993    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558995    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.558998    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559001    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559004    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559006    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559009    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559011    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559014    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559016    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559019    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559022    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559024    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559027    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:34.560797 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559709    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559715    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559719    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559721    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559724    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559727    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559730    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559732    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559735    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559738    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559740    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559744    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559746    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559749    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559752    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559754    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559757    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559759    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559762    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:34.561393 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559765    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559769    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559771    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559774    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559777    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559779    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559782    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559784    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559787    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559789    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559792    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559794    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559797    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559799    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559801    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559804    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559806    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559809    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559811    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559813    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:34.561889 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559816    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559818    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559820    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559823    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559825    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559830    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559833    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559836    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559839    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559841    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559844    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559847    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559851    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559854    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559856    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559859    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559861    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559864    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559866    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559869    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:34.562387 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559871    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559874    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559876    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559879    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559881    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559885    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559888    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559892    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559895    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559899    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559901    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559904    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559907    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559909    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559912    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559914    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559917    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559920    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559923    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:34.562897 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559926    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559929    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559933    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559936    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559938    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559941    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559943    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559946    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559948    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559951    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559953    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559955    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559958    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559960    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559963    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559965    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559968    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559970    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559972    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:34.563380 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559975    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.559977    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560336    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560341    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560343    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560346    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560348    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560351    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560353    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560356    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560358    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560361    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560364    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560367    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560369    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560372    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560375    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560378    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560381    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560383    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:34.563865 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560386    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560388    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560391    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560393    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560397    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560400    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560403    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560406    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560408    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560411    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560415    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560419    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560421    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560426    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560429    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560432    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560435    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560437    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:34.564334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560440    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560442    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560445    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560447    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560450    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560453    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560455    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560458    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560460    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560463    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560466    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560468    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560471    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560474    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560476    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560479    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560481    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560483    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560486    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560488    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:34.564815 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560491    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560493    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560496    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560498    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560501    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560503    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560505    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560508    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560510    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560513    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560515    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560535    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560539    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560543    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560547    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560551    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560554    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560556    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560559    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560561    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:34.565293 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560564    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560567    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560570    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560573    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560578    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560580    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560584    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560586    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560589    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560591    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560594    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560596    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560599    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560602    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560604    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560607    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560609    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560612    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560615    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:34.565839 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560617    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560620    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560622    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.560625    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560701    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560731    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560738    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560742    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560757    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560761    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560765    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560769    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560771    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560774    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560779    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560782    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560785    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560789    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560792    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560795    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560798    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560801    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560805    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560807    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 00:43:34.566334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560811    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560813    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560817    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560821    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560823    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560826    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560829    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560832    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560835    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560838    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560841    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560844    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560849    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560852    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560854    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560857    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560860    2573 flags.go:64] FLAG: --enable-server="true"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560863    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560867    2573 flags.go:64] FLAG: --event-burst="100"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560870    2573 flags.go:64] FLAG: --event-qps="50"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560873    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560876    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560878    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560882    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 00:43:34.566947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560885    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560889    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560892    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560895    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560898    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560900    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560903    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560906    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560909    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560911    2573 flags.go:64] FLAG: --feature-gates=""
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560915    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560918    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560920    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560923    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560926    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560929    2573 flags.go:64] FLAG: --help="false"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560931    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560934    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560937    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560940    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560944    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560947    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560950    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 00:43:34.567544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560953    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560955    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560958    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560961    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560964    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560967    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560969    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560972    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560975    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560978    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560982    2573 flags.go:64] FLAG: --lock-file=""
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560985    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560987    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560990    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560995    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.560998    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561000    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561003    2573 flags.go:64] FLAG: --logging-format="text"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561006    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561009    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561012    2573 flags.go:64] FLAG: --manifest-url=""
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561015    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561018    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561021    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561025    2573 flags.go:64] FLAG: --max-pods="110"
Sep 16 00:43:34.568136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561028    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561030    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561033    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561036    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561041    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561044    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561047    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561054    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561056    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561059    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561062    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561065    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561070    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561073    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561076    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561078    2573 flags.go:64] FLAG: --port="10250"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561081    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561086    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0464c810468d5fef2"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561090    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561092    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561095    2573 flags.go:64] FLAG: --register-node="true"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561098    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561102    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561105    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 16 00:43:34.568778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561107    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561110    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561113    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561117    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561119    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561122    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561125    2573 flags.go:64] FLAG: --runonce="false"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561127    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561130    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561133    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561136    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561139    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561142    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561147    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561149    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561152    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561155    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561158    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561161    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561164    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561166    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561169    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561174    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561177    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561179    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 00:43:34.569356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561184    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561187    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561190    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561193    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561196    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561198    2573 flags.go:64] FLAG: --v="2"
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561202    2573 flags.go:64] FLAG: --version="false"
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561206    2573 flags.go:64] FLAG: --vmodule=""
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561210    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.561213    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561322    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561326    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561329    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561332    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561335    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561337    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561340    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561342    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561345    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561347    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561351    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561354    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:34.570006 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561356    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561359    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561361    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561364    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561366    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561369    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561371    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561374    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561376    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561379    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561382    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561385    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561388    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561391    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561393    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561396    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561399    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561401    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561404    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561409    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:34.570604 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561413    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561416    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561419    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561421    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561424    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561427    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561429    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561432    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561434    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561437    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561441    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561443    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561446    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561449    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561451    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561454    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561456    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561459    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561461    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561464    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:34.571085 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561466    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561469    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561473    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561475    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561478    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561481    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561483    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561486    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561488    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561491    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561493    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561496    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561498    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561500    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561503    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561505    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561508    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561511    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561513    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:34.571577 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561516    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561534    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561536    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561540    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561543    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561545    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561548    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561550    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561553    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561555    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561558    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561560    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561565    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561568    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561571    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561575    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561578    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561580    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561583    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:34.572035 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561586    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561589    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561591    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561593    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561596    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561600    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561604    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561606    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.561609    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.562059    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.569626    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.569640    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.569653    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.569659    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 00:43:34.572485 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569724    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569729    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569734    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569738    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569741    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569744    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569747    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569750    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569753    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569756    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569758    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569761    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569764    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569766    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569769    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569772    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569775    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569778    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569781    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569784    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:34.572844 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569787    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569790    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569792    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569795    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569798    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569800    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569803    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569805    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569808    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569811    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569813    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569816    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569819    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569821    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569824    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569826    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569829    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569831    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569834    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569836    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:34.573352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569839    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569841    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569844    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569847    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569850    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569852    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569855    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569857    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569860    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569863    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569866    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569869    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569872    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569875    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569877    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569880    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569883    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569886    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569890    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:34.573855 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569893    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569896    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569899    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569902    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569906    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569908    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569911    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569914    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569917    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569920    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569923    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569925    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569929    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569932    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569935    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569937    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569940    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569943    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569945    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569948    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:34.574311 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569951    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569953    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569957    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569959    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569961    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569966    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569969    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569972    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569975    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569978    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569980    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569983    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569985    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569988    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569990    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569993    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569995    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.569998    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570001    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:34.574807 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570003    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.570008    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570141    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570145    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570148    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570152    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570155    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570158    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570161    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570163    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570167    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570171    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570174    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570176    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:43:34.575262 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570179    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570182    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570185    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570187    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570190    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570192    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570195    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570198    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570200    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570202    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570205    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570208    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570210    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570213    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570216    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570218    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570221    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570223    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570226    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570228    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:43:34.575616 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570230    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570233    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570235    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570238    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570241    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570243    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570245    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570248    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570250    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570253    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570255    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570258    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570261    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570263    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570265    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570268    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570271    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570273    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570276    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570279    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:43:34.576095 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570281    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570285    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570289    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570291    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570296    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570300    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570302    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570305    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570308    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570311    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570314    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570316    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570319    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570322    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570324    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570327    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570329    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570332    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570334    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:43:34.576586 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570336    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570339    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570342    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570344    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570347    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570349    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570352    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570355    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570357    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570360    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570362    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570364    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570367    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570370    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570372    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570375    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570392    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570396    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570398    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570401    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:43:34.577022 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570404    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570407    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570410    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570412    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570415    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570417    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570420    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:34.570422    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.570427    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.570536    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.572887    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.573944    2573 server.go:1019] "Starting client certificate rotation"
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.574047    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 00:43:34.577497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.574083    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 00:43:34.581113 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.581091    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 00:43:34.582669 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.582650    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 00:43:34.591947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.591931    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 16 00:43:34.597176 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.597159    2573 log.go:25] "Validated CRI v1 image API"
Sep 16 00:43:34.598021 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.598004    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 00:43:34.600847 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.600824    2573 fs.go:135] Filesystem UUIDs: map[73d96822-6a0d-4f90-8329-925cf48ba781:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b15f070e-79b0-45fa-b6a8-548ec368c81a:/dev/nvme0n1p4]
Sep 16 00:43:34.600921 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.600847    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 00:43:34.605791 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.605579    2573 manager.go:216] Machine: {Timestamp:2026-09-16 00:43:34.604314558 +0000 UTC m=+0.307136744 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3137194 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca55ac26d8d21af5499ec90fe516e SystemUUID:ec2ca55a-c26d-8d21-af54-99ec90fe516e BootID:a8eddbc4-85dd-4178-87aa-2be5e14a7eae Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:fb:9d:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:fb:9d:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:2a:cb:86:df:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 00:43:34.605791 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.605784    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 00:43:34.605936 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.605854    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 00:43:34.606060 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.606037    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 00:43:34.606935 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.606910    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 00:43:34.607080 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.606938    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 00:43:34.607673 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.607663    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 00:43:34.607716 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.607677    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 00:43:34.607716 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.607684    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 00:43:34.608393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.608384    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 00:43:34.609437 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.609428    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 00:43:34.612469 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.612457    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 00:43:34.612593 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.612583    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 00:43:34.612638 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.612612    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 00:43:34.612638 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.612629    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 00:43:34.612708 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.612643    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 00:43:34.612708 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.612658    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 16 00:43:34.612708 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.612666    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 00:43:34.613111 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.613098    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 00:43:34.613193 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.613118    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 00:43:34.614931 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.614913    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 00:43:34.617553 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.617536    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 00:43:34.617643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.617575    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 00:43:34.619412 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619396    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 00:43:34.619475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619420    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 00:43:34.619475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619431    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 00:43:34.619475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619440    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 00:43:34.619475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619454    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 00:43:34.619475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619465    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 00:43:34.619645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619488    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 00:43:34.619645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619501    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 00:43:34.619645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619512    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 00:43:34.619645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619539    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 00:43:34.619645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619553    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 00:43:34.619645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.619562    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 00:43:34.620265 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.620254    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 00:43:34.620298 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.620267    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 00:43:34.621947 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.621841    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 00:43:34.622041 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.621897    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 00:43:34.624713 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.624693    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 00:43:34.625291 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.625278    2573 server.go:1268] "Started kubelet"
Sep 16 00:43:34.625350 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.625311    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 00:43:34.625470 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.625412    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 00:43:34.625579 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.625498    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 00:43:34.626078 ip-10-0-131-89 systemd[1]: Started Kubernetes Kubelet.
Sep 16 00:43:34.626917 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.626824    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 00:43:34.626917 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.626911    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 00:43:34.630210 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.629325    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-89.ec2.internal.18d5a66d2366fcb3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-89.ec2.internal,UID:ip-10-0-131-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-89.ec2.internal,},FirstTimestamp:2026-09-16 00:43:34.625238195 +0000 UTC m=+0.328060384,LastTimestamp:2026-09-16 00:43:34.625238195 +0000 UTC m=+0.328060384,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-89.ec2.internal,}"
Sep 16 00:43:34.630301 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.630244    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 00:43:34.632902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.632886    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 00:43:34.632974 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.632902    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 00:43:34.633033 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.632974    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:34.633344 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633329    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 00:43:34.633450 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633408    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 00:43:34.633571 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633553    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 00:43:34.633700 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633683    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 00:43:34.633769 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633703    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 00:43:34.633769 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633712    2573 factory.go:55] Registering systemd factory
Sep 16 00:43:34.633769 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633725    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 16 00:43:34.633973 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633950    2573 factory.go:153] Registering CRI-O factory
Sep 16 00:43:34.633973 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.633968    2573 factory.go:223] Registration of the crio container factory successfully
Sep 16 00:43:34.634087 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.634020    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 00:43:34.634087 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.634037    2573 factory.go:103] Registering Raw factory
Sep 16 00:43:34.634087 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.634052    2573 manager.go:1196] Started watching for new ooms in manager
Sep 16 00:43:34.634423 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.634412    2573 manager.go:349] Starting recovery of all containers
Sep 16 00:43:34.634633 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.634608    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qn5lv"
Sep 16 00:43:34.637355 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.637327    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 00:43:34.637443 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.637412    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 00:43:34.641335 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.641316    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qn5lv"
Sep 16 00:43:34.644313 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.644298    2573 manager.go:354] Recovery completed
Sep 16 00:43:34.657119 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.657107    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:34.658848 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.658827    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:34.658924 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.658862    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:34.658924 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.658874    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:34.659382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.659366    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 00:43:34.659382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.659380    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 00:43:34.659502 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.659395    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 00:43:34.659502 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.659421    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:34.660184 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.660170    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:34.660238 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.660196    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:34.660238 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.660209    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:34.661254 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.661177    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-89.ec2.internal.18d5a66d2567d13d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-89.ec2.internal,UID:ip-10-0-131-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-89.ec2.internal,},FirstTimestamp:2026-09-16 00:43:34.658847037 +0000 UTC m=+0.361669222,LastTimestamp:2026-09-16 00:43:34.658847037 +0000 UTC m=+0.361669222,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-89.ec2.internal,}"
Sep 16 00:43:34.663275 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.663263    2573 policy_none.go:49] "None policy: Start"
Sep 16 00:43:34.663325 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.663279    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 00:43:34.663325 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.663290    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 00:43:34.664814 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.664802    2573 policy_none.go:47] "Start"
Sep 16 00:43:34.706053 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.706039    2573 manager.go:337] "Starting Device Plugin manager"
Sep 16 00:43:34.706188 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.706090    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 00:43:34.706188 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.706103    2573 server.go:87] "Starting device plugin registration server"
Sep 16 00:43:34.706369 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.706356    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 00:43:34.706431 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.706371    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 00:43:34.706482 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.706470    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 00:43:34.706589 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.706570    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 00:43:34.706589 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.706584    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 00:43:34.707015 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.706998    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 00:43:34.707078 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.707052    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:34.762786 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.762759    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 00:43:34.764036 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.764020    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 00:43:34.764136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.764069    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 00:43:34.764136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.764093    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 00:43:34.764136 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.764134    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 00:43:34.766373 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.766347    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:34.807342 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.807251    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:34.808219 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.808205    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:34.808301 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.808236    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:34.808301 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.808252    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:34.808301 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.808278    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.814701 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.814687    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.814767 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.814707    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-89.ec2.internal\": node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:34.827601 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.827584    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:34.865133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.865115    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"]
Sep 16 00:43:34.865198 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.865184    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:34.866419 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.866399    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:34.866504 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.866423    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:34.866504 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.866438    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:34.867702 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.867690    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:34.867815 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.867801    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.867858 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.867842    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:34.868379 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.868364    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:34.868434 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.868399    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:34.868434 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.868369    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:34.868434 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.868415    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:34.868434 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.868429    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:34.868566 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.868446    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:34.869615 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.869602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.869659 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.869626    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:43:34.870258 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.870244    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:43:34.870309 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.870274    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:43:34.870309 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.870289    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:43:34.892212 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.892195    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-89.ec2.internal\" not found" node="ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.896922 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.896907    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-89.ec2.internal\" not found" node="ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.927970 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:34.927949    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:34.934205 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.934187    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16b573fb6b06a96093019490d705bf69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"16b573fb6b06a96093019490d705bf69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.934268 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.934208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b573fb6b06a96093019490d705bf69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"16b573fb6b06a96093019490d705bf69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:34.934268 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:34.934225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b56cb1fea5efc140deb885904593735-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.ec2.internal\" (UID: \"6b56cb1fea5efc140deb885904593735\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.028081 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.028063    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:35.034319 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.034303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16b573fb6b06a96093019490d705bf69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"16b573fb6b06a96093019490d705bf69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.034382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.034329    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b573fb6b06a96093019490d705bf69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"16b573fb6b06a96093019490d705bf69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.034382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.034346    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b56cb1fea5efc140deb885904593735-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.ec2.internal\" (UID: \"6b56cb1fea5efc140deb885904593735\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.034382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.034381    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b56cb1fea5efc140deb885904593735-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.ec2.internal\" (UID: \"6b56cb1fea5efc140deb885904593735\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.034511 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.034420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b573fb6b06a96093019490d705bf69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"16b573fb6b06a96093019490d705bf69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.034511 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.034422    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16b573fb6b06a96093019490d705bf69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"16b573fb6b06a96093019490d705bf69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.128815 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.128747    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:35.196182 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.196156    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.201198 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.201182    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.229078 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.229056    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:35.329560 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.329534    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:35.430057 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.429999    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Sep 16 00:43:35.459701 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.459682    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:35.465726 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.465705    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:35.517323 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.517300    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:35.533378 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.533358    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.545715 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.545698    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 00:43:35.546677 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.546665    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Sep 16 00:43:35.554097 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.554085    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 00:43:35.573185 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.573155    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 00:43:35.573573 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.573292    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 00:43:35.573573 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.573317    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 00:43:35.573573 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.573328    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 00:43:35.573573 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.573328    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 00:43:35.613490 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.613473    2573 apiserver.go:52] "Watching apiserver"
Sep 16 00:43:35.629359 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.629332    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 00:43:35.630968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.630947    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-rp8ph","openshift-network-operator/iptables-alerter-zm5ht","openshift-ovn-kubernetes/ovnkube-node-2s7cl","kube-system/global-pull-secret-syncer-qt698","kube-system/konnectivity-agent-rnb7l","openshift-dns/node-resolver-rhklt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal","openshift-multus/multus-qtzkc","openshift-multus/network-metrics-daemon-nbb66","openshift-network-diagnostics/network-check-target-fp69g","kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2","openshift-cluster-node-tuning-operator/tuned-8qpc4","openshift-image-registry/node-ca-ch47b"]
Sep 16 00:43:35.632275 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.632257    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.633186 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.633163    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 00:43:35.633317 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.633299    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.634357 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.634340    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.634696 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.634672    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 00:43:35.634696 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.634687    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pztrf\""
Sep 16 00:43:35.634899 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.634821    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 00:43:35.634899 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.634883    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 00:43:35.635159 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.635143    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 00:43:35.635249 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.635173    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 00:43:35.635580 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.635563    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 00:43:35.635676 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.635596    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.635676 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.635671    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 00:43:35.635811 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.635791    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 00:43:35.636591 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.636571    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r6rxl\""
Sep 16 00:43:35.636675 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.636580    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 00:43:35.636675 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.636627    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 00:43:35.636675 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.636655    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 00:43:35.636870 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.636854    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 00:43:35.636870 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.636864    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.637055 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637027    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 00:43:35.637151 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637132    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 00:43:35.637213 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637156    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mhfhp\""
Sep 16 00:43:35.637382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637365    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-system-cni-dir\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.637455 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.637455 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637423    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjzk\" (UniqueName: \"kubernetes.io/projected/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-kube-api-access-pmjzk\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.637455 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637432    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:43:35.637455 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-systemd-units\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-var-lib-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637497    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637540    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-ovn\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-node-log\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.637678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgnrw\" (UniqueName: \"kubernetes.io/projected/12358ed4-8f49-4749-bb23-62f5afd0a8c2-kube-api-access-vgnrw\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.637678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-cni-bin\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637662    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n77q2\""
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj6gb\" (UniqueName: \"kubernetes.io/projected/6ced5c7d-da59-4844-baa4-86ac172ce7a7-kube-api-access-jj6gb\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637684    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637724    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-host\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-etc-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-log-socket\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cnibin\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-systemd\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637891    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-cni-netd\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.637964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.637959    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-env-overrides\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638004    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovnkube-script-lib\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638028    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-kubelet\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-slash\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-run-netns\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovnkube-config\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638119    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-serviceca\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.638392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovn-node-metrics-cert\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.638756 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638509    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 00:43:35.638756 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638682    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 00:43:35.638839 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638788    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 00:43:35.638890 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.638870    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q7fk5\""
Sep 16 00:43:35.639115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.639101    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.639620 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.639604    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 00:43:35.639815 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.639803    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fvhzp\""
Sep 16 00:43:35.640056 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.640042    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 00:43:35.640253 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.640241    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:35.640321 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.640306    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:35.640800 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.640783    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 00:43:35.641087 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.641072    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rtjm9\""
Sep 16 00:43:35.641368 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.641355    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:35.641428 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.641415    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:35.642046 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.642031    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 00:43:35.642808 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.642785    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:43:35.644451 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.644371    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 00:38:35 +0000 UTC" deadline="2028-05-04 16:39:18.843010135 +0000 UTC"
Sep 16 00:43:35.644565 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.644452    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14319h55m43.198563033s"
Sep 16 00:43:35.644634 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.644573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.644709 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.644687    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:35.644766 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.644713    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 00:43:35.644766 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.644717    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 00:43:35.644889 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.644873    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qn7nn\""
Sep 16 00:43:35.646115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.646098    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.648579 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.648558    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8sxqn\""
Sep 16 00:43:35.648941 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.648926    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:43:35.649186 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.649158    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 00:43:35.649347 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.649329    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 00:43:35.661002 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.660914    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9ck8x"
Sep 16 00:43:35.666632 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.666615    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9ck8x"
Sep 16 00:43:35.734454 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.734439    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 00:43:35.738421 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738407    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-systemd\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.738478 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.738478 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-cni-netd\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.738478 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738468    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-systemd\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.738582 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738493    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-registration-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.738582 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-systemd\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.738582 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738539    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-kubelet\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.738582 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-cni-netd\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.738694 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-slash\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.738694 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-tmp-dir\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.738694 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738644    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-cni-bin\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.738694 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738670    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b21bf3a4-5aae-4634-b834-08b8e8629924-konnectivity-ca\") pod \"konnectivity-agent-rnb7l\" (UID: \"b21bf3a4-5aae-4634-b834-08b8e8629924\") " pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:43:35.738694 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738670    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-slash\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.738845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738701    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-serviceca\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.738845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-kubernetes\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.738845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738735    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-host\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.738845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-socket-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.738845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-k8s-cni-cncf-io\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.738845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-system-cni-dir\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-systemd-units\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-var-lib-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-systemd-units\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738911    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-system-cni-dir\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-node-log\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738935    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-var-lib-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysctl-d\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738954    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738960    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-node-log\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.738982    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-run\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.739034 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-cni-bin\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-var-lib-kubelet\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739066    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-socket-dir-parent\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739074    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739080    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-cni-bin\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739126    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-serviceca\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739127    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-etc-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-log-socket\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739274    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739280    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-etc-openvswitch\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739298    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-device-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-log-socket\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739368    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvvfv\" (UniqueName: \"kubernetes.io/projected/0e732639-81c7-4971-ba0e-ea1155a44fef-kube-api-access-bvvfv\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739393    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-cni-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739417    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca0e895f-b157-4c49-884c-0bc90b74e724-cni-binary-copy\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739441    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-etc-kubernetes\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.739564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb8de17-005c-4831-a954-9b49d88e804f-host-slash\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739501    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-tuned\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739584    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65lp\" (UniqueName: \"kubernetes.io/projected/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-kube-api-access-q65lp\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-env-overrides\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovnkube-script-lib\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739687    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhx5t\" (UniqueName: \"kubernetes.io/projected/0bb8de17-005c-4831-a954-9b49d88e804f-kube-api-access-vhx5t\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48x7\" (UniqueName: \"kubernetes.io/projected/75d5ce53-37ff-430c-99f7-a1da25718a46-kube-api-access-t48x7\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-cnibin\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739757    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-conf-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739778    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-kubelet\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740349 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-run-netns\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739858    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-kubelet\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovnkube-config\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb8de17-005c-4831-a954-9b49d88e804f-iptables-alerter-script\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-hosts-file\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-cni-multus\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739977    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-host-run-netns\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.739989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-multus-certs\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740022    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovn-node-metrics-cert\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740030    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-env-overrides\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740050    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-etc-selinux\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740075    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-system-cni-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftfnz\" (UniqueName: \"kubernetes.io/projected/ca0e895f-b157-4c49-884c-0bc90b74e724-kube-api-access-ftfnz\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740180    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740204    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovnkube-script-lib\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b21bf3a4-5aae-4634-b834-08b8e8629924-agent-certs\") pod \"konnectivity-agent-rnb7l\" (UID: \"b21bf3a4-5aae-4634-b834-08b8e8629924\") " pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:43:35.740987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjzk\" (UniqueName: \"kubernetes.io/projected/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-kube-api-access-pmjzk\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-ovn\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-modprobe-d\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740329    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysconfig\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740355    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-sys\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740356    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740362    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ced5c7d-da59-4844-baa4-86ac172ce7a7-run-ovn\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740364    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovnkube-config\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740360    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740379    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-sys-fs\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740433    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-daemon-config\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740484    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgnrw\" (UniqueName: \"kubernetes.io/projected/12358ed4-8f49-4749-bb23-62f5afd0a8c2-kube-api-access-vgnrw\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj6gb\" (UniqueName: \"kubernetes.io/projected/6ced5c7d-da59-4844-baa4-86ac172ce7a7-kube-api-access-jj6gb\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-lib-modules\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740600    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-netns\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.741477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740639    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-hostroot\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2v9\" (UniqueName: \"kubernetes.io/projected/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-kube-api-access-gp2v9\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740735    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-host\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740758    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d5ce53-37ff-430c-99f7-a1da25718a46-tmp\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0721d84-496f-43ef-876c-8be3c9dc88ee-kubelet-config\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cnibin\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740836    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysctl-conf\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-host\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740860    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cnibin\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0721d84-496f-43ef-876c-8be3c9dc88ee-dbus\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.741968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.740902    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/12358ed4-8f49-4749-bb23-62f5afd0a8c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.743746 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.743728    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ced5c7d-da59-4844-baa4-86ac172ce7a7-ovn-node-metrics-cert\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.750799 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.750779    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgnrw\" (UniqueName: \"kubernetes.io/projected/12358ed4-8f49-4749-bb23-62f5afd0a8c2-kube-api-access-vgnrw\") pod \"multus-additional-cni-plugins-rp8ph\" (UID: \"12358ed4-8f49-4749-bb23-62f5afd0a8c2\") " pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.751419 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.751397    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjzk\" (UniqueName: \"kubernetes.io/projected/0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e-kube-api-access-pmjzk\") pod \"node-ca-ch47b\" (UID: \"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e\") " pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.751476 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.751446    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj6gb\" (UniqueName: \"kubernetes.io/projected/6ced5c7d-da59-4844-baa4-86ac172ce7a7-kube-api-access-jj6gb\") pod \"ovnkube-node-2s7cl\" (UID: \"6ced5c7d-da59-4844-baa4-86ac172ce7a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:35.841875 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.841728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-run\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.841880    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-run\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.841890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-var-lib-kubelet\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.841914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-socket-dir-parent\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.841939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.841959    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-device-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.841960    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-var-lib-kubelet\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.841981    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvvfv\" (UniqueName: \"kubernetes.io/projected/0e732639-81c7-4971-ba0e-ea1155a44fef-kube-api-access-bvvfv\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-cni-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-socket-dir-parent\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca0e895f-b157-4c49-884c-0bc90b74e724-cni-binary-copy\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.842064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-etc-kubernetes\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-cni-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-device-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842099    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb8de17-005c-4831-a954-9b49d88e804f-host-slash\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842106    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-etc-kubernetes\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-tuned\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842139    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb8de17-005c-4831-a954-9b49d88e804f-host-slash\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65lp\" (UniqueName: \"kubernetes.io/projected/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-kube-api-access-q65lp\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842187    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhx5t\" (UniqueName: \"kubernetes.io/projected/0bb8de17-005c-4831-a954-9b49d88e804f-kube-api-access-vhx5t\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48x7\" (UniqueName: \"kubernetes.io/projected/75d5ce53-37ff-430c-99f7-a1da25718a46-kube-api-access-t48x7\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-cnibin\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-conf-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb8de17-005c-4831-a954-9b49d88e804f-iptables-alerter-script\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-hosts-file\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842380    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-cnibin\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842418    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-cni-multus\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842446    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-hosts-file\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.842662 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.842453    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.842549    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret podName:e0721d84-496f-43ef-876c-8be3c9dc88ee nodeName:}" failed. No retries permitted until 2026-09-16 00:43:36.34250572 +0000 UTC m=+2.045327896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret") pod "global-pull-secret-syncer-qt698" (UID: "e0721d84-496f-43ef-876c-8be3c9dc88ee") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-cni-multus\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842672    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-multus-certs\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842692    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-conf-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842704    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-etc-selinux\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842749    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-multus-certs\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-system-cni-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842788    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-etc-selinux\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842794    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftfnz\" (UniqueName: \"kubernetes.io/projected/ca0e895f-b157-4c49-884c-0bc90b74e724-kube-api-access-ftfnz\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842835    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842859    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842868    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-system-cni-dir\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b21bf3a4-5aae-4634-b834-08b8e8629924-agent-certs\") pod \"konnectivity-agent-rnb7l\" (UID: \"b21bf3a4-5aae-4634-b834-08b8e8629924\") " pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-modprobe-d\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842919    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca0e895f-b157-4c49-884c-0bc90b74e724-cni-binary-copy\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.842940    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:35.843559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysconfig\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842964    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-sys\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.842977    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:43:36.342964483 +0000 UTC m=+2.045786655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.842997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb8de17-005c-4831-a954-9b49d88e804f-iptables-alerter-script\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-sys-fs\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843016    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-sys\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843038    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-sys-fs\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-daemon-config\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-lib-modules\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843088    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysconfig\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-netns\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843103    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-modprobe-d\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843205    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-lib-modules\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843241    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-netns\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843293    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-hostroot\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-hostroot\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843364    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2v9\" (UniqueName: \"kubernetes.io/projected/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-kube-api-access-gp2v9\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:35.844623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d5ce53-37ff-430c-99f7-a1da25718a46-tmp\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843410    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0721d84-496f-43ef-876c-8be3c9dc88ee-kubelet-config\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysctl-conf\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0721d84-496f-43ef-876c-8be3c9dc88ee-dbus\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843493    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-systemd\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-registration-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-kubelet\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca0e895f-b157-4c49-884c-0bc90b74e724-multus-daemon-config\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-tmp-dir\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-cni-bin\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b21bf3a4-5aae-4634-b834-08b8e8629924-konnectivity-ca\") pod \"konnectivity-agent-rnb7l\" (UID: \"b21bf3a4-5aae-4634-b834-08b8e8629924\") " pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-kubernetes\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-host\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-socket-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-k8s-cni-cncf-io\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysctl-d\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843777    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-kubelet\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843862    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-systemd\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.845445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysctl-conf\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-registration-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843931    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0721d84-496f-43ef-876c-8be3c9dc88ee-kubelet-config\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843937    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0721d84-496f-43ef-876c-8be3c9dc88ee-dbus\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-var-lib-cni-bin\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.843984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-host\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.844032    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-kubernetes\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.844073    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca0e895f-b157-4c49-884c-0bc90b74e724-host-run-k8s-cni-cncf-io\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.844137    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-tmp-dir\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.844177    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e732639-81c7-4971-ba0e-ea1155a44fef-socket-dir\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.844251    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-sysctl-d\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.844374    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b21bf3a4-5aae-4634-b834-08b8e8629924-konnectivity-ca\") pod \"konnectivity-agent-rnb7l\" (UID: \"b21bf3a4-5aae-4634-b834-08b8e8629924\") " pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.845099    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75d5ce53-37ff-430c-99f7-a1da25718a46-etc-tuned\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.845375    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b21bf3a4-5aae-4634-b834-08b8e8629924-agent-certs\") pod \"konnectivity-agent-rnb7l\" (UID: \"b21bf3a4-5aae-4634-b834-08b8e8629924\") " pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:43:35.846108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.845966    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d5ce53-37ff-430c-99f7-a1da25718a46-tmp\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.853244 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.853223    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhx5t\" (UniqueName: \"kubernetes.io/projected/0bb8de17-005c-4831-a954-9b49d88e804f-kube-api-access-vhx5t\") pod \"iptables-alerter-zm5ht\" (UID: \"0bb8de17-005c-4831-a954-9b49d88e804f\") " pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:35.853470 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.853455    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:35.853551 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.853475    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:35.853551 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.853488    2573 projected.go:196] Error preparing data for projected volume kube-api-access-c4bcb for pod openshift-network-diagnostics/network-check-target-fp69g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:35.853635 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:35.853558    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb podName:d8c93b16-7ddb-4dce-91a5-966c7ee70e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:36.353544384 +0000 UTC m=+2.056366560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c4bcb" (UniqueName: "kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb") pod "network-check-target-fp69g" (UID: "d8c93b16-7ddb-4dce-91a5-966c7ee70e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:35.854755 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.854731    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftfnz\" (UniqueName: \"kubernetes.io/projected/ca0e895f-b157-4c49-884c-0bc90b74e724-kube-api-access-ftfnz\") pod \"multus-qtzkc\" (UID: \"ca0e895f-b157-4c49-884c-0bc90b74e724\") " pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:35.855946 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.855924    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvvfv\" (UniqueName: \"kubernetes.io/projected/0e732639-81c7-4971-ba0e-ea1155a44fef-kube-api-access-bvvfv\") pod \"aws-ebs-csi-driver-node-q62z2\" (UID: \"0e732639-81c7-4971-ba0e-ea1155a44fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:35.856032 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.855925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2v9\" (UniqueName: \"kubernetes.io/projected/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-kube-api-access-gp2v9\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:35.856032 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.855962    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48x7\" (UniqueName: \"kubernetes.io/projected/75d5ce53-37ff-430c-99f7-a1da25718a46-kube-api-access-t48x7\") pod \"tuned-8qpc4\" (UID: \"75d5ce53-37ff-430c-99f7-a1da25718a46\") " pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:35.856356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.856342    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65lp\" (UniqueName: \"kubernetes.io/projected/93bd702c-41a7-4c5b-b7a2-24428b0e7a18-kube-api-access-q65lp\") pod \"node-resolver-rhklt\" (UID: \"93bd702c-41a7-4c5b-b7a2-24428b0e7a18\") " pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:35.926812 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:35.926776    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b56cb1fea5efc140deb885904593735.slice/crio-9f20dd4edec84959669380f602bb75c42b63c868e1dcb2c86b1d1e47b52fd0ad WatchSource:0}: Error finding container 9f20dd4edec84959669380f602bb75c42b63c868e1dcb2c86b1d1e47b52fd0ad: Status 404 returned error can't find the container with id 9f20dd4edec84959669380f602bb75c42b63c868e1dcb2c86b1d1e47b52fd0ad
Sep 16 00:43:35.927190 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:35.927164    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b573fb6b06a96093019490d705bf69.slice/crio-ea4023d7e1b6ced58ff065885cce40eee311a36ab794d7f5885e2a6c26487302 WatchSource:0}: Error finding container ea4023d7e1b6ced58ff065885cce40eee311a36ab794d7f5885e2a6c26487302: Status 404 returned error can't find the container with id ea4023d7e1b6ced58ff065885cce40eee311a36ab794d7f5885e2a6c26487302
Sep 16 00:43:35.931235 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.931221    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:43:35.960769 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.960723    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rp8ph"
Sep 16 00:43:35.967787 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:35.967769    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12358ed4_8f49_4749_bb23_62f5afd0a8c2.slice/crio-c0fe0aafcecc971a401db9aadaa1eb9db979c38f3e9fd3a4170184ab8dde7c33 WatchSource:0}: Error finding container c0fe0aafcecc971a401db9aadaa1eb9db979c38f3e9fd3a4170184ab8dde7c33: Status 404 returned error can't find the container with id c0fe0aafcecc971a401db9aadaa1eb9db979c38f3e9fd3a4170184ab8dde7c33
Sep 16 00:43:35.978662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.978637    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ch47b"
Sep 16 00:43:35.985088 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:35.985071    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9eb6d9_33b8_4ded_b33e_5f1facefbf9e.slice/crio-31de40304c28756053bcb65a641630e1eeeb3ae33ec1ba9db3864f21a6d9f1f3 WatchSource:0}: Error finding container 31de40304c28756053bcb65a641630e1eeeb3ae33ec1ba9db3864f21a6d9f1f3: Status 404 returned error can't find the container with id 31de40304c28756053bcb65a641630e1eeeb3ae33ec1ba9db3864f21a6d9f1f3
Sep 16 00:43:35.998599 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:35.998582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:43:36.004983 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:36.004966    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ced5c7d_da59_4844_baa4_86ac172ce7a7.slice/crio-ac5d98395c31baa425878eaacd75737ae94a613e4e3dac1b333c637af9e70720 WatchSource:0}: Error finding container ac5d98395c31baa425878eaacd75737ae94a613e4e3dac1b333c637af9e70720: Status 404 returned error can't find the container with id ac5d98395c31baa425878eaacd75737ae94a613e4e3dac1b333c637af9e70720
Sep 16 00:43:36.010833 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.010816    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8qpc4"
Sep 16 00:43:36.018104 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:36.018085    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d5ce53_37ff_430c_99f7_a1da25718a46.slice/crio-3906722356d3c305517fb8ba9141e80064fc9e506329bc53da0b79b3d2560869 WatchSource:0}: Error finding container 3906722356d3c305517fb8ba9141e80064fc9e506329bc53da0b79b3d2560869: Status 404 returned error can't find the container with id 3906722356d3c305517fb8ba9141e80064fc9e506329bc53da0b79b3d2560869
Sep 16 00:43:36.030593 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.030576    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2"
Sep 16 00:43:36.036562 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.036544    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rhklt"
Sep 16 00:43:36.037224 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:36.037207    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e732639_81c7_4971_ba0e_ea1155a44fef.slice/crio-5b1cfcdcb9c6832387ac65a1d2d8cc954941eab7581f3a12b567c74ae140983b WatchSource:0}: Error finding container 5b1cfcdcb9c6832387ac65a1d2d8cc954941eab7581f3a12b567c74ae140983b: Status 404 returned error can't find the container with id 5b1cfcdcb9c6832387ac65a1d2d8cc954941eab7581f3a12b567c74ae140983b
Sep 16 00:43:36.043379 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:36.043360    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bd702c_41a7_4c5b_b7a2_24428b0e7a18.slice/crio-2ad8a4da423f90cc7a4c6d3e8fa1600c48db32f708af75ddcff08949049f7389 WatchSource:0}: Error finding container 2ad8a4da423f90cc7a4c6d3e8fa1600c48db32f708af75ddcff08949049f7389: Status 404 returned error can't find the container with id 2ad8a4da423f90cc7a4c6d3e8fa1600c48db32f708af75ddcff08949049f7389
Sep 16 00:43:36.072138 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.072120    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qtzkc"
Sep 16 00:43:36.078954 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:36.078931    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0e895f_b157_4c49_884c_0bc90b74e724.slice/crio-2053089af711133bef246482906245e86b5ffbca9edd2f5feca06075e8cd0096 WatchSource:0}: Error finding container 2053089af711133bef246482906245e86b5ffbca9edd2f5feca06075e8cd0096: Status 404 returned error can't find the container with id 2053089af711133bef246482906245e86b5ffbca9edd2f5feca06075e8cd0096
Sep 16 00:43:36.090631 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.090607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:43:36.096226 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.096200    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zm5ht"
Sep 16 00:43:36.097754 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:36.097731    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21bf3a4_5aae_4634_b834_08b8e8629924.slice/crio-90c24aa2221486261a5df7787815380352c0a235bb550febe64d6de91a4916fb WatchSource:0}: Error finding container 90c24aa2221486261a5df7787815380352c0a235bb550febe64d6de91a4916fb: Status 404 returned error can't find the container with id 90c24aa2221486261a5df7787815380352c0a235bb550febe64d6de91a4916fb
Sep 16 00:43:36.104040 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:43:36.104008    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb8de17_005c_4831_a954_9b49d88e804f.slice/crio-c80f2f01026fc0312026b92e4dbca8470a3b08ca87db5942bece7d9c0cfe2ae4 WatchSource:0}: Error finding container c80f2f01026fc0312026b92e4dbca8470a3b08ca87db5942bece7d9c0cfe2ae4: Status 404 returned error can't find the container with id c80f2f01026fc0312026b92e4dbca8470a3b08ca87db5942bece7d9c0cfe2ae4
Sep 16 00:43:36.105641 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.105620    2573 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-zm5ht_openshift-network-operator(0bb8de17-005c-4831-a954-9b49d88e804f): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 16 00:43:36.105731 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.105654    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-zm5ht" podUID="0bb8de17-005c-4831-a954-9b49d88e804f"
Sep 16 00:43:36.347885 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.347808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:36.347885 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.347844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:36.348065 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.347962    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:36.348065 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.347962    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:36.348065 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.348025    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret podName:e0721d84-496f-43ef-876c-8be3c9dc88ee nodeName:}" failed. No retries permitted until 2026-09-16 00:43:37.348004305 +0000 UTC m=+3.050826496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret") pod "global-pull-secret-syncer-qt698" (UID: "e0721d84-496f-43ef-876c-8be3c9dc88ee") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:36.348065 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.348044    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:43:37.348033288 +0000 UTC m=+3.050855466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:36.448358 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.448316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:36.460911 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.460881    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:36.460911 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.460917    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:36.461116 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.460938    2573 projected.go:196] Error preparing data for projected volume kube-api-access-c4bcb for pod openshift-network-diagnostics/network-check-target-fp69g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:36.461116 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.461009    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb podName:d8c93b16-7ddb-4dce-91a5-966c7ee70e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:37.460980706 +0000 UTC m=+3.163802884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4bcb" (UniqueName: "kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb") pod "network-check-target-fp69g" (UID: "d8c93b16-7ddb-4dce-91a5-966c7ee70e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:36.669800 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.669542    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 00:38:36 +0000 UTC" deadline="2028-03-26 19:50:19.3081394 +0000 UTC"
Sep 16 00:43:36.669800 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.669608    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13387h6m42.638536229s"
Sep 16 00:43:36.768303 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.767511    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:36.768303 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.767663    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:36.800325 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.800262    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rnb7l" event={"ID":"b21bf3a4-5aae-4634-b834-08b8e8629924","Type":"ContainerStarted","Data":"90c24aa2221486261a5df7787815380352c0a235bb550febe64d6de91a4916fb"}
Sep 16 00:43:36.809544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.809508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2" event={"ID":"0e732639-81c7-4971-ba0e-ea1155a44fef","Type":"ContainerStarted","Data":"5b1cfcdcb9c6832387ac65a1d2d8cc954941eab7581f3a12b567c74ae140983b"}
Sep 16 00:43:36.818912 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.818881    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"ac5d98395c31baa425878eaacd75737ae94a613e4e3dac1b333c637af9e70720"}
Sep 16 00:43:36.828333 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.828292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ch47b" event={"ID":"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e","Type":"ContainerStarted","Data":"31de40304c28756053bcb65a641630e1eeeb3ae33ec1ba9db3864f21a6d9f1f3"}
Sep 16 00:43:36.839578 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.839549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" event={"ID":"12358ed4-8f49-4749-bb23-62f5afd0a8c2","Type":"ContainerStarted","Data":"c0fe0aafcecc971a401db9aadaa1eb9db979c38f3e9fd3a4170184ab8dde7c33"}
Sep 16 00:43:36.846047 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.846019    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal" event={"ID":"16b573fb6b06a96093019490d705bf69","Type":"ContainerStarted","Data":"ea4023d7e1b6ced58ff065885cce40eee311a36ab794d7f5885e2a6c26487302"}
Sep 16 00:43:36.857540 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.857512    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zm5ht" event={"ID":"0bb8de17-005c-4831-a954-9b49d88e804f","Type":"ContainerStarted","Data":"c80f2f01026fc0312026b92e4dbca8470a3b08ca87db5942bece7d9c0cfe2ae4"}
Sep 16 00:43:36.865643 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:36.865570    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0444a9580f430c7c6b52f78667a80a82198c825805f1f2004fdfdf9aed2b95f5\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-zm5ht" podUID="0bb8de17-005c-4831-a954-9b49d88e804f"
Sep 16 00:43:36.880875 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.880845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qtzkc" event={"ID":"ca0e895f-b157-4c49-884c-0bc90b74e724","Type":"ContainerStarted","Data":"2053089af711133bef246482906245e86b5ffbca9edd2f5feca06075e8cd0096"}
Sep 16 00:43:36.896514 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.896480    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rhklt" event={"ID":"93bd702c-41a7-4c5b-b7a2-24428b0e7a18","Type":"ContainerStarted","Data":"2ad8a4da423f90cc7a4c6d3e8fa1600c48db32f708af75ddcff08949049f7389"}
Sep 16 00:43:36.905556 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.905526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8qpc4" event={"ID":"75d5ce53-37ff-430c-99f7-a1da25718a46","Type":"ContainerStarted","Data":"3906722356d3c305517fb8ba9141e80064fc9e506329bc53da0b79b3d2560869"}
Sep 16 00:43:36.917522 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:36.917492    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal" event={"ID":"6b56cb1fea5efc140deb885904593735","Type":"ContainerStarted","Data":"9f20dd4edec84959669380f602bb75c42b63c868e1dcb2c86b1d1e47b52fd0ad"}
Sep 16 00:43:37.135354 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.134808    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:37.219322 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.219285    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:37.356549 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.356510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:37.356763 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.356587    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:37.356763 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.356740    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:37.356885 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.356801    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:43:39.356781183 +0000 UTC m=+5.059603361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:37.357260 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.357191    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:37.357260 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.357244    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret podName:e0721d84-496f-43ef-876c-8be3c9dc88ee nodeName:}" failed. No retries permitted until 2026-09-16 00:43:39.357227472 +0000 UTC m=+5.060049650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret") pod "global-pull-secret-syncer-qt698" (UID: "e0721d84-496f-43ef-876c-8be3c9dc88ee") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:37.375837 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.375808    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:37.558837 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.558755    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:37.559012 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.558923    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:37.559012 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.558945    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:37.559012 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.558958    2573 projected.go:196] Error preparing data for projected volume kube-api-access-c4bcb for pod openshift-network-diagnostics/network-check-target-fp69g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:37.559187 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.559013    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb podName:d8c93b16-7ddb-4dce-91a5-966c7ee70e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:39.558995631 +0000 UTC m=+5.261817809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4bcb" (UniqueName: "kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb") pod "network-check-target-fp69g" (UID: "d8c93b16-7ddb-4dce-91a5-966c7ee70e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:37.669964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.669915    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 00:38:36 +0000 UTC" deadline="2028-06-25 00:37:35.428683342 +0000 UTC"
Sep 16 00:43:37.669964 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.669961    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15551h53m57.758727049s"
Sep 16 00:43:37.765394 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.764629    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:37.765394 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.764768    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:37.765394 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:37.765184    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:37.765394 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:37.765303    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:38.762262 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:38.762151    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:43:38.767735 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:38.767261    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:38.767735 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:38.767384    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:39.376633 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:39.376597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:39.376837 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:39.376653    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:39.376837 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.376810    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:39.376958 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.376872    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:43:43.376851743 +0000 UTC m=+9.079673932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:39.377395 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.377291    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:39.377395 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.377345    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret podName:e0721d84-496f-43ef-876c-8be3c9dc88ee nodeName:}" failed. No retries permitted until 2026-09-16 00:43:43.377330432 +0000 UTC m=+9.080152614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret") pod "global-pull-secret-syncer-qt698" (UID: "e0721d84-496f-43ef-876c-8be3c9dc88ee") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:39.579679 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:39.579640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:39.579868 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.579793    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:39.579868 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.579813    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:39.579868 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.579826    2573 projected.go:196] Error preparing data for projected volume kube-api-access-c4bcb for pod openshift-network-diagnostics/network-check-target-fp69g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:39.580039 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.579882    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb podName:d8c93b16-7ddb-4dce-91a5-966c7ee70e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:43.579864062 +0000 UTC m=+9.282686254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4bcb" (UniqueName: "kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb") pod "network-check-target-fp69g" (UID: "d8c93b16-7ddb-4dce-91a5-966c7ee70e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:39.765218 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:39.765131    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:39.765218 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:39.765153    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:39.765794 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.765267    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:39.765794 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:39.765669    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:40.766541 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:40.766071    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:40.766541 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:40.766201    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:41.766058 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:41.765385    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:41.766058 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:41.765527    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:41.766058 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:41.765939    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:41.766058 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:41.766019    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:42.765173 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:42.765138    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:42.765691 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:42.765269    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:43.411868 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:43.411582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:43.411868 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:43.411644    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:43.411868 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.411760    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:43.411868 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.411760    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:43.411868 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.411824    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:43:51.41180595 +0000 UTC m=+17.114628140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:43.411868 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.411841    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret podName:e0721d84-496f-43ef-876c-8be3c9dc88ee nodeName:}" failed. No retries permitted until 2026-09-16 00:43:51.41183196 +0000 UTC m=+17.114654142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret") pod "global-pull-secret-syncer-qt698" (UID: "e0721d84-496f-43ef-876c-8be3c9dc88ee") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:43.613442 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:43.613348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:43.613651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.613555    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:43.613651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.613598    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:43.613651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.613613    2573 projected.go:196] Error preparing data for projected volume kube-api-access-c4bcb for pod openshift-network-diagnostics/network-check-target-fp69g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:43.613838 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.613668    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb podName:d8c93b16-7ddb-4dce-91a5-966c7ee70e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:43:51.613654056 +0000 UTC m=+17.316476230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4bcb" (UniqueName: "kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb") pod "network-check-target-fp69g" (UID: "d8c93b16-7ddb-4dce-91a5-966c7ee70e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:43.765346 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:43.764633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:43.765346 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.764773    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:43.765346 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:43.765175    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:43.765346 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:43.765296    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:44.765787 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:44.765388    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:44.765787 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:44.765468    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:45.765352 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:45.765321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:45.765570 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:45.765476    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:45.765857 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:45.765321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:45.766186 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:45.765963    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:46.764801 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:46.764764    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:46.764973 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:46.764922    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:47.764986 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:47.764953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:47.765429 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:47.765102    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:47.765429 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:47.765161    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:47.765429 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:47.765294    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:48.765297 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:48.765220    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:48.765738 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:48.765337    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:49.764645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:49.764613    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:49.764809 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:49.764623    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:49.764900 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:49.764808    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:49.764900 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:49.764833    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:50.767855 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:50.766697    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:50.768333 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:50.767935    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:51.476935 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:51.476892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:51.477174 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:51.476959    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:51.477174 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.477056    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:51.477174 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.477111    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:51.477174 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.477136    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret podName:e0721d84-496f-43ef-876c-8be3c9dc88ee nodeName:}" failed. No retries permitted until 2026-09-16 00:44:07.47711586 +0000 UTC m=+33.179938045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret") pod "global-pull-secret-syncer-qt698" (UID: "e0721d84-496f-43ef-876c-8be3c9dc88ee") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:43:51.477174 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.477157    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:44:07.477143692 +0000 UTC m=+33.179965870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:43:51.678374 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:51.678352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:51.678470 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.678458    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:43:51.678507 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.678476    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:43:51.678507 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.678485    2573 projected.go:196] Error preparing data for projected volume kube-api-access-c4bcb for pod openshift-network-diagnostics/network-check-target-fp69g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:51.678588 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.678547    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb podName:d8c93b16-7ddb-4dce-91a5-966c7ee70e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:44:07.678534981 +0000 UTC m=+33.381357167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4bcb" (UniqueName: "kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb") pod "network-check-target-fp69g" (UID: "d8c93b16-7ddb-4dce-91a5-966c7ee70e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:43:51.764793 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:51.764737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:51.764793 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:51.764767    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:51.764919 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.764846    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:51.764985 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:51.764968    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:52.765050 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:52.765016    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:52.765490 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:52.765153    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:53.764485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:53.764451    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:53.764485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:53.764485    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:53.764736 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:53.764580    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:53.764736 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:53.764711    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:54.767594 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:54.767563    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:54.768041 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:54.767672    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:55.765234 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:55.765194    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:55.765394 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:55.765297    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:55.765394 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:55.765364    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:55.765503 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:55.765469    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:55.997361 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:55.997010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8qpc4" event={"ID":"75d5ce53-37ff-430c-99f7-a1da25718a46","Type":"ContainerStarted","Data":"1a4641a600642fea4642c5714b579b3251a32c1ab1463df4a1e7d90358d54db0"}
Sep 16 00:43:56.002026 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:56.001993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal" event={"ID":"6b56cb1fea5efc140deb885904593735","Type":"ContainerStarted","Data":"c5ff4307b704d00a57ef311849e7c1f6cd1da93ff299aecd4cb4913e9007f684"}
Sep 16 00:43:56.027233 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:56.026564    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal" podStartSLOduration=20.026505217 podStartE2EDuration="20.026505217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:56.026463301 +0000 UTC m=+21.729285495" watchObservedRunningTime="2026-09-16 00:43:56.026505217 +0000 UTC m=+21.729327411"
Sep 16 00:43:56.027233 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:56.026751    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8qpc4" podStartSLOduration=1.225073515 podStartE2EDuration="21.02673965s" totalImagesPullingTime="19.801666135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:56.012810506 +0000 UTC m=+21.715632703" watchObservedRunningTime="2026-09-16 00:43:56.02673965 +0000 UTC m=+21.729561846"
Sep 16 00:43:56.768077 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:56.767896    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:56.768213 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:56.768146    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:57.010712 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.010679    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rnb7l" event={"ID":"b21bf3a4-5aae-4634-b834-08b8e8629924","Type":"ContainerStarted","Data":"dc772b63f71401107c0765c7fc402f3a29efd6c446e27ba668a388dbcace9828"}
Sep 16 00:43:57.012362 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.012338    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2" event={"ID":"0e732639-81c7-4971-ba0e-ea1155a44fef","Type":"ContainerStarted","Data":"4fd762a1fa294f1a53f50055a8a79208921ca66251e4f2ab7e181b26c9570c4c"}
Sep 16 00:43:57.015941 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.015918    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"4ab6a536aa292d09a6057784b19f98c4c6d1c256a9aa19d388d42c676fe3f400"}
Sep 16 00:43:57.016048 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.015997    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"1ac77d726f767dfe78643dcea3b31950cb881b746f4e953b73703531e42153c3"}
Sep 16 00:43:57.016112 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.016064    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"24a18c843accb9279851a2a962e211b9698b8fcf870248c9e980048a71c3faf2"}
Sep 16 00:43:57.016165 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.016116    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"f95ed45e618293e9b88c592dc004a118841bdf30cf75424d73de07465dcf5209"}
Sep 16 00:43:57.016221 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.016166    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"7e672ddf396462e14f068a3b37e34c87c173065aa3a82f6371d0dc91fb7675c3"}
Sep 16 00:43:57.016221 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.016215    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"708199649edfbef712c28080b8179820f44c0c3c42eb725f204dc0ef3dc3a175"}
Sep 16 00:43:57.017685 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.017661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ch47b" event={"ID":"0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e","Type":"ContainerStarted","Data":"6d307cd4be8bccc8ace39d40969f021385ff5c996145b93d50b5ffe576d5050f"}
Sep 16 00:43:57.019346 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.019322    2573 generic.go:356] "Generic (PLEG): container finished" podID="12358ed4-8f49-4749-bb23-62f5afd0a8c2" containerID="5d94bc3b9e89cf8b9656c10d83d1772bd452c20ed5a1ca972fa20d68b44fd183" exitCode=0
Sep 16 00:43:57.019457 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.019396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" event={"ID":"12358ed4-8f49-4749-bb23-62f5afd0a8c2","Type":"ContainerDied","Data":"5d94bc3b9e89cf8b9656c10d83d1772bd452c20ed5a1ca972fa20d68b44fd183"}
Sep 16 00:43:57.021292 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.021268    2573 generic.go:356] "Generic (PLEG): container finished" podID="16b573fb6b06a96093019490d705bf69" containerID="20f4a4e90f428c677b0b579ceb497e1ba7951c48a18e5066334fdabfd1d29f9b" exitCode=0
Sep 16 00:43:57.021375 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.021331    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal" event={"ID":"16b573fb6b06a96093019490d705bf69","Type":"ContainerDied","Data":"20f4a4e90f428c677b0b579ceb497e1ba7951c48a18e5066334fdabfd1d29f9b"}
Sep 16 00:43:57.023289 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.023264    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qtzkc" event={"ID":"ca0e895f-b157-4c49-884c-0bc90b74e724","Type":"ContainerStarted","Data":"28b808ecec8bcb983ac14d97c9257be2d4ee9632cf6fce4da4eac3831a70a4e5"}
Sep 16 00:43:57.024842 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.024787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rhklt" event={"ID":"93bd702c-41a7-4c5b-b7a2-24428b0e7a18","Type":"ContainerStarted","Data":"264f760d15471f1838be34e8046b32caea4a1e21962056c08da4bdb29a84cecd"}
Sep 16 00:43:57.038453 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.038416    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rnb7l" podStartSLOduration=1.3185927849999999 podStartE2EDuration="21.03840537s" totalImagesPullingTime="19.719812585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:57.024499801 +0000 UTC m=+22.727321996" watchObservedRunningTime="2026-09-16 00:43:57.03840537 +0000 UTC m=+22.741227564"
Sep 16 00:43:57.067907 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.067856    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ch47b" podStartSLOduration=6.396220908 podStartE2EDuration="22.06783734s" totalImagesPullingTime="15.671616432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:57.067777641 +0000 UTC m=+22.770599835" watchObservedRunningTime="2026-09-16 00:43:57.06783734 +0000 UTC m=+22.770659535"
Sep 16 00:43:57.082117 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.082075    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qtzkc" podStartSLOduration=1.2684714590000001 podStartE2EDuration="21.082060685s" totalImagesPullingTime="19.813589226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:57.081801636 +0000 UTC m=+22.784623821" watchObservedRunningTime="2026-09-16 00:43:57.082060685 +0000 UTC m=+22.784882885"
Sep 16 00:43:57.099225 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.099186    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rhklt" podStartSLOduration=2.323377023 podStartE2EDuration="22.09917153s" totalImagesPullingTime="19.775794507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:57.098976246 +0000 UTC m=+22.801798444" watchObservedRunningTime="2026-09-16 00:43:57.09917153 +0000 UTC m=+22.801993726"
Sep 16 00:43:57.639872 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.639779    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 00:43:57.719106 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.719016    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T00:43:57.639799768Z","UUID":"733b5c53-d789-4f06-bbbf-23c40e49605b","Handler":null,"Name":"","Endpoint":""}
Sep 16 00:43:57.720797 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.720657    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 00:43:57.720891 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.720809    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 00:43:57.765198 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.765145    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:57.765287 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:57.765238    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:43:57.765335 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:57.765294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:57.765390 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:57.765375    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:58.029465 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:58.029385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal" event={"ID":"16b573fb6b06a96093019490d705bf69","Type":"ContainerStarted","Data":"229f3e6896e3b9084091c2bd59a38221763db96a6103531d274e58b382480b04"}
Sep 16 00:43:58.031000 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:58.030959    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zm5ht" event={"ID":"0bb8de17-005c-4831-a954-9b49d88e804f","Type":"ContainerStarted","Data":"6b6597d4a3cebbb245cc5b8f4de0d1514c3884fc462d2f494706e0f6a33ad378"}
Sep 16 00:43:58.033124 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:58.033094    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2" event={"ID":"0e732639-81c7-4971-ba0e-ea1155a44fef","Type":"ContainerStarted","Data":"081d40926c3c16d3e6c57df9e692c34a3aca0da45ef0390cd33822fb4c5564c0"}
Sep 16 00:43:58.058720 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:58.058672    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zm5ht" podStartSLOduration=2.280788847 podStartE2EDuration="22.05865706s" totalImagesPullingTime="19.777868213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-16 00:43:58.058103121 +0000 UTC m=+23.760925317" watchObservedRunningTime="2026-09-16 00:43:58.05865706 +0000 UTC m=+23.761479255"
Sep 16 00:43:58.059023 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:58.058999    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal" podStartSLOduration=22.058992737 podStartE2EDuration="22.058992737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:58.043883685 +0000 UTC m=+23.746705881" watchObservedRunningTime="2026-09-16 00:43:58.058992737 +0000 UTC m=+23.761814931"
Sep 16 00:43:58.765461 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:58.765277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:43:58.765461 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:58.765420    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:43:59.036715 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:59.036634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2" event={"ID":"0e732639-81c7-4971-ba0e-ea1155a44fef","Type":"ContainerStarted","Data":"f268be4c20cf9d05619946a015408e171e5ec5120aa5cb232a264808a96e9fc7"}
Sep 16 00:43:59.039935 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:59.039911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"341d1c83aaa477e58c078c5ef375bfde84216ae4dd621bd772b7fcb84f5f0e7c"}
Sep 16 00:43:59.765236 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:59.765205    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:43:59.765406 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:43:59.765205    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:43:59.765406 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:59.765315    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:43:59.765553 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:43:59.765396    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:44:00.764446 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:00.764406    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:00.764877 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:00.764534    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:44:01.393394 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:01.393360    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:44:01.393986 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:01.393965    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:44:01.408711 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:01.408670    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q62z2" podStartSLOduration=4.462216262 podStartE2EDuration="26.408648884s" totalImagesPullingTime="21.946432622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:59.052076368 +0000 UTC m=+24.754898563" watchObservedRunningTime="2026-09-16 00:44:01.408648884 +0000 UTC m=+27.111471087"
Sep 16 00:44:01.764752 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:01.764579    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:01.765129 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:01.764586    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:44:01.765129 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:01.764837    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:44:01.765129 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:01.764919    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:44:02.049664 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:02.049571    2573 generic.go:356] "Generic (PLEG): container finished" podID="12358ed4-8f49-4749-bb23-62f5afd0a8c2" containerID="3c38b324e4b9041b078874c37f18c4dd2087a3fc1a94620a4091a7635263f45a" exitCode=0
Sep 16 00:44:02.049664 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:02.049613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" event={"ID":"12358ed4-8f49-4749-bb23-62f5afd0a8c2","Type":"ContainerDied","Data":"3c38b324e4b9041b078874c37f18c4dd2087a3fc1a94620a4091a7635263f45a"}
Sep 16 00:44:02.053783 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:02.053760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" event={"ID":"6ced5c7d-da59-4844-baa4-86ac172ce7a7","Type":"ContainerStarted","Data":"e203f2eb85617e68c1f02a6890fc2410baf94968ea19dca7335f4e54a804cba2"}
Sep 16 00:44:02.053981 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:02.053960    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:44:02.054403 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:02.054383    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rnb7l"
Sep 16 00:44:02.090604 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:02.090555    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl" podStartSLOduration=7.222217613 podStartE2EDuration="27.090539329s" totalImagesPullingTime="19.868321716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:02.090129082 +0000 UTC m=+27.792951279" watchObservedRunningTime="2026-09-16 00:44:02.090539329 +0000 UTC m=+27.793361516"
Sep 16 00:44:02.768008 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:02.767978    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:02.768565 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:02.768102    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:44:03.058669 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.058596    2573 generic.go:356] "Generic (PLEG): container finished" podID="12358ed4-8f49-4749-bb23-62f5afd0a8c2" containerID="8709275554dd9677d53db9cfb32048bd0d6384e1fa751fbd062e5f207fb9ac60" exitCode=0
Sep 16 00:44:03.058796 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.058675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" event={"ID":"12358ed4-8f49-4749-bb23-62f5afd0a8c2","Type":"ContainerDied","Data":"8709275554dd9677d53db9cfb32048bd0d6384e1fa751fbd062e5f207fb9ac60"}
Sep 16 00:44:03.059617 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.059371    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:44:03.059617 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.059445    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:44:03.059617 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.059481    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:44:03.076794 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.076765    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:44:03.077163 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.077144    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:44:03.484005 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.483928    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qt698"]
Sep 16 00:44:03.484156 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.484076    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:03.484221 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:03.484182    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:44:03.486778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.486751    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nbb66"]
Sep 16 00:44:03.486876 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.486860    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:44:03.486966 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:03.486950    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:44:03.498133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.498108    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fp69g"]
Sep 16 00:44:03.498255 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:03.498207    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:03.498317 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:03.498300    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:44:04.063624 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:04.063350    2573 generic.go:356] "Generic (PLEG): container finished" podID="12358ed4-8f49-4749-bb23-62f5afd0a8c2" containerID="6d63de288387dd382587743a7d9ce03eac59bdd631884be05ac391f935d6988e" exitCode=0
Sep 16 00:44:04.064323 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:04.063437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" event={"ID":"12358ed4-8f49-4749-bb23-62f5afd0a8c2","Type":"ContainerDied","Data":"6d63de288387dd382587743a7d9ce03eac59bdd631884be05ac391f935d6988e"}
Sep 16 00:44:04.769920 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:04.769889    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:04.770096 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:04.770026    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:44:05.764961 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:05.764890    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:44:05.765351 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:05.764889    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:05.765351 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:05.765024    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:44:05.765351 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:05.765087    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:44:06.764881 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:06.764846    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:06.765032 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:06.764977    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp69g" podUID="d8c93b16-7ddb-4dce-91a5-966c7ee70e28"
Sep 16 00:44:07.491774 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:07.491696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:07.491774 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:07.491750    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:44:07.491991 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.491853    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:44:07.491991 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.491858    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:44:07.491991 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.491926    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret podName:e0721d84-496f-43ef-876c-8be3c9dc88ee nodeName:}" failed. No retries permitted until 2026-09-16 00:44:39.491907071 +0000 UTC m=+65.194729258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret") pod "global-pull-secret-syncer-qt698" (UID: "e0721d84-496f-43ef-876c-8be3c9dc88ee") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:44:07.491991 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.491947    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:44:39.491937153 +0000 UTC m=+65.194759330 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:44:07.693229 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:07.693198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:07.693380 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.693358    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:44:07.693380 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.693376    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:44:07.693495 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.693388    2573 projected.go:196] Error preparing data for projected volume kube-api-access-c4bcb for pod openshift-network-diagnostics/network-check-target-fp69g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:44:07.693495 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.693440    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb podName:d8c93b16-7ddb-4dce-91a5-966c7ee70e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:44:39.693422306 +0000 UTC m=+65.396244498 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4bcb" (UniqueName: "kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb") pod "network-check-target-fp69g" (UID: "d8c93b16-7ddb-4dce-91a5-966c7ee70e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:44:07.764714 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:07.764645    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:44:07.764714 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:07.764657    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:07.764877 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.764743    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:44:07.764877 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:07.764864    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qt698" podUID="e0721d84-496f-43ef-876c-8be3c9dc88ee"
Sep 16 00:44:08.569672 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.569643    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeReady"
Sep 16 00:44:08.569979 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.569824    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 00:44:08.617845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.617822    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7m8gh"]
Sep 16 00:44:08.632200 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.632175    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hf8lr"]
Sep 16 00:44:08.632321 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.632295    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.634308 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.634286    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 00:44:08.634419 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.634343    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n5l29\""
Sep 16 00:44:08.635086 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.635064    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 00:44:08.648486 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.648465    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hf8lr"]
Sep 16 00:44:08.648602 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.648496    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7m8gh"]
Sep 16 00:44:08.648667 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.648611    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hf8lr"
Sep 16 00:44:08.650730 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.650713    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 00:44:08.650825 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.650802    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tzjjl\""
Sep 16 00:44:08.651004 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.650986    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 00:44:08.651103 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.651021    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 00:44:08.701346 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.701326    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.701447 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.701393    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-config-volume\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.701484 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.701446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-tmp-dir\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.701484 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.701475    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhq4m\" (UniqueName: \"kubernetes.io/projected/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-kube-api-access-nhq4m\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.764804 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.764781    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:08.767396 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.767376    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 00:44:08.767396 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.767390    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ssxtd\""
Sep 16 00:44:08.767581 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.767450    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 00:44:08.802504 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.802482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-tmp-dir\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.802614 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.802509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhq4m\" (UniqueName: \"kubernetes.io/projected/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-kube-api-access-nhq4m\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.802614 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.802551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e064abea-ce8c-4fad-b012-d23a5fbf95ab-cert\") pod \"ingress-canary-hf8lr\" (UID: \"e064abea-ce8c-4fad-b012-d23a5fbf95ab\") " pod="openshift-ingress-canary/ingress-canary-hf8lr"
Sep 16 00:44:08.802614 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.802572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2hj\" (UniqueName: \"kubernetes.io/projected/e064abea-ce8c-4fad-b012-d23a5fbf95ab-kube-api-access-8t2hj\") pod \"ingress-canary-hf8lr\" (UID: \"e064abea-ce8c-4fad-b012-d23a5fbf95ab\") " pod="openshift-ingress-canary/ingress-canary-hf8lr"
Sep 16 00:44:08.802614 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.802597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.802830 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:08.802713    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:08.802830 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.802760    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-config-volume\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.802830 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:08.802791    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:44:09.302761582 +0000 UTC m=+35.005583757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:08.802946 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.802833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-tmp-dir\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.803149 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.803134    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-config-volume\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.812280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.812252    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhq4m\" (UniqueName: \"kubernetes.io/projected/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-kube-api-access-nhq4m\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:08.903052 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.902993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e064abea-ce8c-4fad-b012-d23a5fbf95ab-cert\") pod \"ingress-canary-hf8lr\" (UID: \"e064abea-ce8c-4fad-b012-d23a5fbf95ab\") " pod="openshift-ingress-canary/ingress-canary-hf8lr"
Sep 16 00:44:08.903052 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.903030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2hj\" (UniqueName: \"kubernetes.io/projected/e064abea-ce8c-4fad-b012-d23a5fbf95ab-kube-api-access-8t2hj\") pod \"ingress-canary-hf8lr\" (UID: \"e064abea-ce8c-4fad-b012-d23a5fbf95ab\") " pod="openshift-ingress-canary/ingress-canary-hf8lr"
Sep 16 00:44:08.905656 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.905636    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e064abea-ce8c-4fad-b012-d23a5fbf95ab-cert\") pod \"ingress-canary-hf8lr\" (UID: \"e064abea-ce8c-4fad-b012-d23a5fbf95ab\") " pod="openshift-ingress-canary/ingress-canary-hf8lr"
Sep 16 00:44:08.911055 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.911037    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2hj\" (UniqueName: \"kubernetes.io/projected/e064abea-ce8c-4fad-b012-d23a5fbf95ab-kube-api-access-8t2hj\") pod \"ingress-canary-hf8lr\" (UID: \"e064abea-ce8c-4fad-b012-d23a5fbf95ab\") " pod="openshift-ingress-canary/ingress-canary-hf8lr"
Sep 16 00:44:08.986991 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:08.986970    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hf8lr"
Sep 16 00:44:09.078966 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.078765    2573 generic.go:356] "Generic (PLEG): container finished" podID="12358ed4-8f49-4749-bb23-62f5afd0a8c2" containerID="86fb32e0e80ffe30a28e8dba66138d323b40f4b061609e15da62a272a75f0a10" exitCode=0
Sep 16 00:44:09.079126 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.078930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" event={"ID":"12358ed4-8f49-4749-bb23-62f5afd0a8c2","Type":"ContainerDied","Data":"86fb32e0e80ffe30a28e8dba66138d323b40f4b061609e15da62a272a75f0a10"}
Sep 16 00:44:09.287907 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.287879    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hf8lr"]
Sep 16 00:44:09.303254 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:44:09.303227    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode064abea_ce8c_4fad_b012_d23a5fbf95ab.slice/crio-3cb00de58a2c4c64b2fb1de5dbe79ab8c22aa36adbecebcda9c3134f765f9d8c WatchSource:0}: Error finding container 3cb00de58a2c4c64b2fb1de5dbe79ab8c22aa36adbecebcda9c3134f765f9d8c: Status 404 returned error can't find the container with id 3cb00de58a2c4c64b2fb1de5dbe79ab8c22aa36adbecebcda9c3134f765f9d8c
Sep 16 00:44:09.306902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.306884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:09.306982 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:09.306970    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:09.307025 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:09.307014    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:44:10.307001432 +0000 UTC m=+36.009823605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:09.764636 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.764614    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:44:09.765099 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.764641    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:09.767026 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.767001    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 00:44:09.767026 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.767007    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6btht\""
Sep 16 00:44:09.767026 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:09.767021    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 00:44:10.085386 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:10.085312    2573 generic.go:356] "Generic (PLEG): container finished" podID="12358ed4-8f49-4749-bb23-62f5afd0a8c2" containerID="55e45232df4c72a11a7e299fc5a57ff50987b8ed2e26d4f07dcf0f320e026900" exitCode=0
Sep 16 00:44:10.085549 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:10.085396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" event={"ID":"12358ed4-8f49-4749-bb23-62f5afd0a8c2","Type":"ContainerDied","Data":"55e45232df4c72a11a7e299fc5a57ff50987b8ed2e26d4f07dcf0f320e026900"}
Sep 16 00:44:10.086952 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:10.086772    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hf8lr" event={"ID":"e064abea-ce8c-4fad-b012-d23a5fbf95ab","Type":"ContainerStarted","Data":"3cb00de58a2c4c64b2fb1de5dbe79ab8c22aa36adbecebcda9c3134f765f9d8c"}
Sep 16 00:44:10.314254 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:10.314227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:10.314407 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:10.314386    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:10.314473 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:10.314462    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:44:12.314440461 +0000 UTC m=+38.017262648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:11.093869 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:11.093829    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" event={"ID":"12358ed4-8f49-4749-bb23-62f5afd0a8c2","Type":"ContainerStarted","Data":"7852be0d287bb471a9f8855e6cc89b96f37c9fe3c38d554595f2be9dc8de8607"}
Sep 16 00:44:11.115688 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:11.115642    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rp8ph" podStartSLOduration=6.134268272 podStartE2EDuration="36.115628445s" totalImagesPullingTime="29.981360173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:11.11495367 +0000 UTC m=+36.817775866" watchObservedRunningTime="2026-09-16 00:44:11.115628445 +0000 UTC m=+36.818450641"
Sep 16 00:44:12.326772 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:12.326692    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:12.327221 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:12.326798    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:12.327221 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:12.326849    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:44:16.326836433 +0000 UTC m=+42.029658605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:13.099302 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:13.099238    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hf8lr" event={"ID":"e064abea-ce8c-4fad-b012-d23a5fbf95ab","Type":"ContainerStarted","Data":"c390905a1374ad98c7de4f75bafc1c0e68722eef2082b4e04b43658abc84309f"}
Sep 16 00:44:13.114327 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:13.114289    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hf8lr" podStartSLOduration=2.393209944 podStartE2EDuration="5.114273028s" totalImagesPullingTime="2.721063084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:13.11427358 +0000 UTC m=+38.817095776" watchObservedRunningTime="2026-09-16 00:44:13.114273028 +0000 UTC m=+38.817095217"
Sep 16 00:44:16.352980 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:16.352784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:16.352980 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:16.352933    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:16.353362 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:16.353038    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:44:24.353019995 +0000 UTC m=+50.055842172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:24.404968 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:24.404926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:24.405376 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:24.405057    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:24.405376 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:24.405117    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:44:40.405103527 +0000 UTC m=+66.107925703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:35.077245 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:35.077219    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s7cl"
Sep 16 00:44:39.503173 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.503141    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:39.503635 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.503179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:44:39.505807 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.505789    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 00:44:39.505874 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.505856    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 00:44:39.513571 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:39.513548    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 00:44:39.513672 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:39.513608    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:45:43.513591794 +0000 UTC m=+129.216413971 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : secret "metrics-daemon-secret" not found
Sep 16 00:44:39.516280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.516265    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0721d84-496f-43ef-876c-8be3c9dc88ee-original-pull-secret\") pod \"global-pull-secret-syncer-qt698\" (UID: \"e0721d84-496f-43ef-876c-8be3c9dc88ee\") " pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:39.704404 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.704379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:39.706906 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.706887    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 00:44:39.717550 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.717510    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 00:44:39.728431 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.728410    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4bcb\" (UniqueName: \"kubernetes.io/projected/d8c93b16-7ddb-4dce-91a5-966c7ee70e28-kube-api-access-c4bcb\") pod \"network-check-target-fp69g\" (UID: \"d8c93b16-7ddb-4dce-91a5-966c7ee70e28\") " pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:39.778533 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.778474    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qt698"
Sep 16 00:44:39.974768 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.974736    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ssxtd\""
Sep 16 00:44:39.983763 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:39.983740    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:40.077191 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:40.077120    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qt698"]
Sep 16 00:44:40.081372 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:44:40.081348    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0721d84_496f_43ef_876c_8be3c9dc88ee.slice/crio-0f44c384bfe56ad03797e52bf32950a6b51b70a76ef17cbd5efcef3bd40a9c1e WatchSource:0}: Error finding container 0f44c384bfe56ad03797e52bf32950a6b51b70a76ef17cbd5efcef3bd40a9c1e: Status 404 returned error can't find the container with id 0f44c384bfe56ad03797e52bf32950a6b51b70a76ef17cbd5efcef3bd40a9c1e
Sep 16 00:44:40.161011 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:40.160980    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qt698" event={"ID":"e0721d84-496f-43ef-876c-8be3c9dc88ee","Type":"ContainerStarted","Data":"0f44c384bfe56ad03797e52bf32950a6b51b70a76ef17cbd5efcef3bd40a9c1e"}
Sep 16 00:44:40.270145 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:40.270121    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fp69g"]
Sep 16 00:44:40.273023 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:44:40.273002    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c93b16_7ddb_4dce_91a5_966c7ee70e28.slice/crio-22651a240ade43f831966938d76107b4e831ebb53c70d9cb265db508b2d10e78 WatchSource:0}: Error finding container 22651a240ade43f831966938d76107b4e831ebb53c70d9cb265db508b2d10e78: Status 404 returned error can't find the container with id 22651a240ade43f831966938d76107b4e831ebb53c70d9cb265db508b2d10e78
Sep 16 00:44:40.408848 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:40.408782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:44:40.409032 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:40.408900    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:44:40.409032 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:44:40.408957    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:45:12.408939873 +0000 UTC m=+98.111762048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:44:41.165852 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:41.165815    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fp69g" event={"ID":"d8c93b16-7ddb-4dce-91a5-966c7ee70e28","Type":"ContainerStarted","Data":"22651a240ade43f831966938d76107b4e831ebb53c70d9cb265db508b2d10e78"}
Sep 16 00:44:45.178110 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:45.178017    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qt698" event={"ID":"e0721d84-496f-43ef-876c-8be3c9dc88ee","Type":"ContainerStarted","Data":"1e786b9db269c9fdef757288b10fb0a346b16848224da335781619af68108627"}
Sep 16 00:44:45.179577 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:45.179554    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fp69g" event={"ID":"d8c93b16-7ddb-4dce-91a5-966c7ee70e28","Type":"ContainerStarted","Data":"bfdafd1836568aac601bb0fec02b77bfc30bc72d140c1f80feeb8d6045af0978"}
Sep 16 00:44:45.179687 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:45.179676    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:44:45.190755 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:44:45.190710    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qt698" podStartSLOduration=64.383740683 podStartE2EDuration="1m9.190698711s" totalImagesPullingTime="4.806958028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:45.190145887 +0000 UTC m=+70.892968081" watchObservedRunningTime="2026-09-16 00:44:45.190698711 +0000 UTC m=+70.893520905"
Sep 16 00:45:12.418352 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:45:12.418305    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:45:12.418861 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:45:12.418454    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:45:12.418861 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:45:12.418536    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:46:16.418502491 +0000 UTC m=+162.121324666 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:45:16.184669 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:45:16.184639    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fp69g"
Sep 16 00:45:16.198325 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:45:16.198273    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fp69g" podStartSLOduration=95.586191441 podStartE2EDuration="1m40.198258937s" totalImagesPullingTime="4.612067496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:45.2064256 +0000 UTC m=+70.909247794" watchObservedRunningTime="2026-09-16 00:45:16.198258937 +0000 UTC m=+101.901081156"
Sep 16 00:45:43.607404 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:45:43.607352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:45:43.607905 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:45:43.607496    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 00:45:43.607905 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:45:43.607577    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs podName:5527c19f-cd6a-4cd6-adc4-0bb7e70f140d nodeName:}" failed. No retries permitted until 2026-09-16 00:47:45.607561529 +0000 UTC m=+251.310383719 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs") pod "network-metrics-daemon-nbb66" (UID: "5527c19f-cd6a-4cd6-adc4-0bb7e70f140d") : secret "metrics-daemon-secret" not found
Sep 16 00:46:11.682829 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:46:11.682774    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7m8gh" podUID="59b8ae0f-6819-4e75-8d12-27c5156bcd9d"
Sep 16 00:46:12.388216 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:46:12.388185    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:46:12.774597 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:46:12.774558    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nbb66" podUID="5527c19f-cd6a-4cd6-adc4-0bb7e70f140d"
Sep 16 00:46:16.515361 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:46:16.515314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:46:16.515778 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:46:16.515433    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:46:16.515778 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:46:16.515484    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls podName:59b8ae0f-6819-4e75-8d12-27c5156bcd9d nodeName:}" failed. No retries permitted until 2026-09-16 00:48:18.51546973 +0000 UTC m=+284.218291903 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls") pod "dns-default-7m8gh" (UID: "59b8ae0f-6819-4e75-8d12-27c5156bcd9d") : secret "dns-default-metrics-tls" not found
Sep 16 00:46:25.765022 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:46:25.764929    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:47:15.832918 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.832886    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp"]
Sep 16 00:47:15.834566 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.834542    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"]
Sep 16 00:47:15.834662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.834643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp"
Sep 16 00:47:15.836142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.836121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:15.847564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.847543    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8mbgp\""
Sep 16 00:47:15.847645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.847608    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 00:47:15.847885 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:15.847858    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"network-diagnostics-dockercfg-m5sjg\" is forbidden: User \"system:node:ip-10-0-131-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-network-diagnostics\": no relationship found between node 'ip-10-0-131-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-m5sjg\"" type="*v1.Secret"
Sep 16 00:47:15.848067 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.848046    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 00:47:15.848146 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.848107    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:47:15.851183 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.851164    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"]
Sep 16 00:47:15.852941 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.852926    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-74867654d6-ccpxx"]
Sep 16 00:47:15.853072 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.853058    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:15.854725 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.854707    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:15.855172 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.855155    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 00:47:15.855257 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.855216    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 00:47:15.855882 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.855867    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 00:47:15.856461 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.856445    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:47:15.856802 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.856788    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 00:47:15.856849 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.856793    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d8ls9\""
Sep 16 00:47:15.856849 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.856819    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 00:47:15.857386 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.857371    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 00:47:15.857445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.857427    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 00:47:15.857565 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.857549    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 00:47:15.857648 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.857590    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kzrzm\""
Sep 16 00:47:15.857648 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.857598    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 00:47:15.865397 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.865375    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"]
Sep 16 00:47:15.874610 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.874588    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp"]
Sep 16 00:47:15.879087 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.879067    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"]
Sep 16 00:47:15.884026 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.884006    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74867654d6-ccpxx"]
Sep 16 00:47:15.896889 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.896869    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzcps\" (UniqueName: \"kubernetes.io/projected/1d87713d-f966-444a-8cf5-3bc5310210ed-kube-api-access-lzcps\") pod \"network-check-source-6f8b768455-ss7zp\" (UID: \"1d87713d-f966-444a-8cf5-3bc5310210ed\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp"
Sep 16 00:47:15.896983 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.896898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:15.896983 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.896932    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptcs\" (UniqueName: \"kubernetes.io/projected/066e3b1a-2974-404d-81e0-c30a22b787ba-kube-api-access-sptcs\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:15.926747 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.926724    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"]
Sep 16 00:47:15.930798 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.930669    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:15.933022 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.933005    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 00:47:15.933812 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.933796    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 00:47:15.933896 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.933816    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-95779\""
Sep 16 00:47:15.933896 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.933870    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 00:47:15.934012 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.933910    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 00:47:15.935498 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.935481    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5db64bf567-9gr2t"]
Sep 16 00:47:15.937445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.937429    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4"]
Sep 16 00:47:15.937598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.937579    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:15.939395 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.939375    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4"
Sep 16 00:47:15.940229 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.940198    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 00:47:15.940229 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.940225    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 00:47:15.940374 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.940278    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l2glv\""
Sep 16 00:47:15.940488 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.940473    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 00:47:15.940663 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.940642    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"]
Sep 16 00:47:15.941950 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.941931    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 00:47:15.942289 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.942269    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:47:15.942399 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.942324    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dzfxs\""
Sep 16 00:47:15.948890 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.948752    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 00:47:15.949474 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.949454    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4"]
Sep 16 00:47:15.952793 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.952458    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5db64bf567-9gr2t"]
Sep 16 00:47:15.997602 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997584    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8fj\" (UniqueName: \"kubernetes.io/projected/eee257f5-5e28-4785-ab61-b9d403220ec4-kube-api-access-gr8fj\") pod \"volume-data-source-validator-86695b7cc7-pvhr4\" (UID: \"eee257f5-5e28-4785-ab61-b9d403220ec4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4"
Sep 16 00:47:15.997713 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997609    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/226d6ebd-5397-4939-b861-33f1d5da4a10-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:15.997713 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:15.997713 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997649    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-default-certificate\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:15.997879 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997724    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-installation-pull-secrets\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:15.997879 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzcps\" (UniqueName: \"kubernetes.io/projected/1d87713d-f966-444a-8cf5-3bc5310210ed-kube-api-access-lzcps\") pod \"network-check-source-6f8b768455-ss7zp\" (UID: \"1d87713d-f966-444a-8cf5-3bc5310210ed\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp"
Sep 16 00:47:15.997879 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-certificates\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:15.997879 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997823    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:15.997879 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99772ee8-b1af-4d1c-acd4-55ea52a1a114-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997891    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9f8s\" (UniqueName: \"kubernetes.io/projected/99772ee8-b1af-4d1c-acd4-55ea52a1a114-kube-api-access-d9f8s\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsgsj\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-kube-api-access-vsgsj\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sptcs\" (UniqueName: \"kubernetes.io/projected/066e3b1a-2974-404d-81e0-c30a22b787ba-kube-api-access-sptcs\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.997993    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsng4\" (UniqueName: \"kubernetes.io/projected/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-kube-api-access-lsng4\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998053    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-stats-auth\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:15.998115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998108    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-image-registry-private-configuration\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:15.998475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzqq\" (UniqueName: \"kubernetes.io/projected/226d6ebd-5397-4939-b861-33f1d5da4a10-kube-api-access-jmzqq\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:15.998475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99772ee8-b1af-4d1c-acd4-55ea52a1a114-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:15.998475 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:15.998208    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:47:15.998475 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:15.998269    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls podName:066e3b1a-2974-404d-81e0-c30a22b787ba nodeName:}" failed. No retries permitted until 2026-09-16 00:47:16.498251171 +0000 UTC m=+222.201073348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c8977" (UID: "066e3b1a-2974-404d-81e0-c30a22b787ba") : secret "samples-operator-tls" not found
Sep 16 00:47:15.998475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998210    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-ca-trust-extracted\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:15.998475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-trusted-ca\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:15.998475 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:15.998340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-bound-sa-token\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.006301 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.006282    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzcps\" (UniqueName: \"kubernetes.io/projected/1d87713d-f966-444a-8cf5-3bc5310210ed-kube-api-access-lzcps\") pod \"network-check-source-6f8b768455-ss7zp\" (UID: \"1d87713d-f966-444a-8cf5-3bc5310210ed\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp"
Sep 16 00:47:16.009042 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.009019    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptcs\" (UniqueName: \"kubernetes.io/projected/066e3b1a-2974-404d-81e0-c30a22b787ba-kube-api-access-sptcs\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:16.099382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099332    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgsj\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-kube-api-access-vsgsj\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.099382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.099382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099377    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsng4\" (UniqueName: \"kubernetes.io/projected/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-kube-api-access-lsng4\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.099598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-stats-auth\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.099598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-image-registry-private-configuration\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.099598 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.099463    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:47:16.099598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099480    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzqq\" (UniqueName: \"kubernetes.io/projected/226d6ebd-5397-4939-b861-33f1d5da4a10-kube-api-access-jmzqq\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:16.099598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99772ee8-b1af-4d1c-acd4-55ea52a1a114-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:16.099598 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.099514    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:16.599501531 +0000 UTC m=+222.302323703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : secret "router-metrics-certs-default" not found
Sep 16 00:47:16.099598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-ca-trust-extracted\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.099598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-trusted-ca\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099625    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-bound-sa-token\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8fj\" (UniqueName: \"kubernetes.io/projected/eee257f5-5e28-4785-ab61-b9d403220ec4-kube-api-access-gr8fj\") pod \"volume-data-source-validator-86695b7cc7-pvhr4\" (UID: \"eee257f5-5e28-4785-ab61-b9d403220ec4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/226d6ebd-5397-4939-b861-33f1d5da4a10-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099754    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-default-certificate\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099783    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-installation-pull-secrets\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099815    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-certificates\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099868    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99772ee8-b1af-4d1c-acd4-55ea52a1a114-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099911    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9f8s\" (UniqueName: \"kubernetes.io/projected/99772ee8-b1af-4d1c-acd4-55ea52a1a114-kube-api-access-d9f8s\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:16.099988 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.099985    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-ca-trust-extracted\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.100243    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:16.600228644 +0000 UTC m=+222.303050840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : configmap references non-existent config key: service-ca.crt
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.100579    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.100630    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls podName:226d6ebd-5397-4939-b861-33f1d5da4a10 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:16.600614405 +0000 UTC m=+222.303436595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4r5r7" (UID: "226d6ebd-5397-4939-b861-33f1d5da4a10") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.100674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-certificates\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.100774    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.100787    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db64bf567-9gr2t: secret "image-registry-tls" not found
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.100829    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls podName:3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e nodeName:}" failed. No retries permitted until 2026-09-16 00:47:16.600812981 +0000 UTC m=+222.303635169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls") pod "image-registry-5db64bf567-9gr2t" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e") : secret "image-registry-tls" not found
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.100831    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99772ee8-b1af-4d1c-acd4-55ea52a1a114-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:16.100942 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.100850    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/226d6ebd-5397-4939-b861-33f1d5da4a10-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:16.101493 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.101469    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-trusted-ca\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.101976 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.101957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99772ee8-b1af-4d1c-acd4-55ea52a1a114-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:16.102771 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.102747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-image-registry-private-configuration\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.102876 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.102854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-default-certificate\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.103088 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.103071    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-installation-pull-secrets\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.103273 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.103254    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-stats-auth\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.108289 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.108262    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzqq\" (UniqueName: \"kubernetes.io/projected/226d6ebd-5397-4939-b861-33f1d5da4a10-kube-api-access-jmzqq\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:16.108482 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.108459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsgsj\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-kube-api-access-vsgsj\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.108815 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.108798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9f8s\" (UniqueName: \"kubernetes.io/projected/99772ee8-b1af-4d1c-acd4-55ea52a1a114-kube-api-access-d9f8s\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9982t\" (UID: \"99772ee8-b1af-4d1c-acd4-55ea52a1a114\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:16.108902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.108854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-bound-sa-token\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.108902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.108885    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsng4\" (UniqueName: \"kubernetes.io/projected/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-kube-api-access-lsng4\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.109000 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.108967    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8fj\" (UniqueName: \"kubernetes.io/projected/eee257f5-5e28-4785-ab61-b9d403220ec4-kube-api-access-gr8fj\") pod \"volume-data-source-validator-86695b7cc7-pvhr4\" (UID: \"eee257f5-5e28-4785-ab61-b9d403220ec4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4"
Sep 16 00:47:16.162794 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.162773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"
Sep 16 00:47:16.255749 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.255712    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4"
Sep 16 00:47:16.427130 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.427058    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t"]
Sep 16 00:47:16.430031 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:16.430010    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99772ee8_b1af_4d1c_acd4_55ea52a1a114.slice/crio-e99247b737c66b3d1e9b3112a5cc650c6f647ff6937d6a72f8360b765c8ea172 WatchSource:0}: Error finding container e99247b737c66b3d1e9b3112a5cc650c6f647ff6937d6a72f8360b765c8ea172: Status 404 returned error can't find the container with id e99247b737c66b3d1e9b3112a5cc650c6f647ff6937d6a72f8360b765c8ea172
Sep 16 00:47:16.503213 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.503180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:16.503347 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.503320    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:47:16.503412 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.503393    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls podName:066e3b1a-2974-404d-81e0-c30a22b787ba nodeName:}" failed. No retries permitted until 2026-09-16 00:47:17.503373376 +0000 UTC m=+223.206195564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c8977" (UID: "066e3b1a-2974-404d-81e0-c30a22b787ba") : secret "samples-operator-tls" not found
Sep 16 00:47:16.524426 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.524402    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4"]
Sep 16 00:47:16.527353 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:16.527332    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee257f5_5e28_4785_ab61_b9d403220ec4.slice/crio-adeea745d2ea7ab496fba36debb3310865c31198864fc1d63d3c7495251804b6 WatchSource:0}: Error finding container adeea745d2ea7ab496fba36debb3310865c31198864fc1d63d3c7495251804b6: Status 404 returned error can't find the container with id adeea745d2ea7ab496fba36debb3310865c31198864fc1d63d3c7495251804b6
Sep 16 00:47:16.537249 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.537222    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4" event={"ID":"eee257f5-5e28-4785-ab61-b9d403220ec4","Type":"ContainerStarted","Data":"adeea745d2ea7ab496fba36debb3310865c31198864fc1d63d3c7495251804b6"}
Sep 16 00:47:16.538255 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.538235    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t" event={"ID":"99772ee8-b1af-4d1c-acd4-55ea52a1a114","Type":"ContainerStarted","Data":"e99247b737c66b3d1e9b3112a5cc650c6f647ff6937d6a72f8360b765c8ea172"}
Sep 16 00:47:16.603948 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.603920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.604072 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.603962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:16.604072 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.603983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:16.604072 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:16.604037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:16.604279 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.604077    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:17.60406108 +0000 UTC m=+223.306883252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : configmap references non-existent config key: service-ca.crt
Sep 16 00:47:16.604279 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.604094    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:47:16.604279 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.604115    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db64bf567-9gr2t: secret "image-registry-tls" not found
Sep 16 00:47:16.604279 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.604127    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:16.604279 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.604136    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:47:16.604279 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.604173    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls podName:3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e nodeName:}" failed. No retries permitted until 2026-09-16 00:47:17.604155423 +0000 UTC m=+223.306977611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls") pod "image-registry-5db64bf567-9gr2t" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e") : secret "image-registry-tls" not found
Sep 16 00:47:16.604279 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.604190    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls podName:226d6ebd-5397-4939-b861-33f1d5da4a10 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:17.604181333 +0000 UTC m=+223.307003521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4r5r7" (UID: "226d6ebd-5397-4939-b861-33f1d5da4a10") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:16.604279 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:16.604207    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:17.604198792 +0000 UTC m=+223.307021131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : secret "router-metrics-certs-default" not found
Sep 16 00:47:17.145570 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.145511    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Sep 16 00:47:17.146005 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.145599    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp"
Sep 16 00:47:17.184436 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.184405    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-m5sjg\""
Sep 16 00:47:17.427703 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.427584    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp"]
Sep 16 00:47:17.512719 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.512681    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:17.512884 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.512855    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:47:17.512946 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.512936    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls podName:066e3b1a-2974-404d-81e0-c30a22b787ba nodeName:}" failed. No retries permitted until 2026-09-16 00:47:19.512915058 +0000 UTC m=+225.215737235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c8977" (UID: "066e3b1a-2974-404d-81e0-c30a22b787ba") : secret "samples-operator-tls" not found
Sep 16 00:47:17.585826 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:17.585785    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d87713d_f966_444a_8cf5_3bc5310210ed.slice/crio-879fda8c820e2aa5aec5d0d0eb91dfb4ddb49a8c3ff0245a8c48be54c81ce5cc WatchSource:0}: Error finding container 879fda8c820e2aa5aec5d0d0eb91dfb4ddb49a8c3ff0245a8c48be54c81ce5cc: Status 404 returned error can't find the container with id 879fda8c820e2aa5aec5d0d0eb91dfb4ddb49a8c3ff0245a8c48be54c81ce5cc
Sep 16 00:47:17.613129 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.613065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:17.613129 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.613116    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:17.613231 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.613167    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:47:17.613231 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.613181    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db64bf567-9gr2t: secret "image-registry-tls" not found
Sep 16 00:47:17.613231 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.613219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:17.613231 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.613231    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls podName:3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e nodeName:}" failed. No retries permitted until 2026-09-16 00:47:19.613216959 +0000 UTC m=+225.316039135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls") pod "image-registry-5db64bf567-9gr2t" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e") : secret "image-registry-tls" not found
Sep 16 00:47:17.613392 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.613263    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:47:17.613392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:17.613285    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:17.613392 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.613294    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:17.613392 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.613321    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:19.613303154 +0000 UTC m=+225.316125345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : secret "router-metrics-certs-default" not found
Sep 16 00:47:17.613392 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.613345    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls podName:226d6ebd-5397-4939-b861-33f1d5da4a10 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:19.613328391 +0000 UTC m=+225.316150583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4r5r7" (UID: "226d6ebd-5397-4939-b861-33f1d5da4a10") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:17.613392 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:17.613387    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:19.613373949 +0000 UTC m=+225.316196138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : configmap references non-existent config key: service-ca.crt
Sep 16 00:47:18.545814 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:18.545731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4" event={"ID":"eee257f5-5e28-4785-ab61-b9d403220ec4","Type":"ContainerStarted","Data":"4c51f9b822ceb5a74ecb215a01795a139783b394edbc0eb3bd39d8b6bcca55d3"}
Sep 16 00:47:18.547663 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:18.547636    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp" event={"ID":"1d87713d-f966-444a-8cf5-3bc5310210ed","Type":"ContainerStarted","Data":"761760f0900a8fe2c256b27ce0049fb5187540abb146b72b3d81e0abe71ff487"}
Sep 16 00:47:18.547805 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:18.547682    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp" event={"ID":"1d87713d-f966-444a-8cf5-3bc5310210ed","Type":"ContainerStarted","Data":"879fda8c820e2aa5aec5d0d0eb91dfb4ddb49a8c3ff0245a8c48be54c81ce5cc"}
Sep 16 00:47:18.549193 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:18.549166    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t" event={"ID":"99772ee8-b1af-4d1c-acd4-55ea52a1a114","Type":"ContainerStarted","Data":"ae486b558d57ebc0f7e8888a2236faa04c4bc49c57bbac01b55c3dc2ea59635f"}
Sep 16 00:47:18.560764 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:18.560724    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pvhr4" podStartSLOduration=2.477335685 podStartE2EDuration="3.560710995s" totalImagesPullingTime="1.08337531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:18.559911463 +0000 UTC m=+224.262733659" watchObservedRunningTime="2026-09-16 00:47:18.560710995 +0000 UTC m=+224.263533190"
Sep 16 00:47:18.576699 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:18.576665    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t" podStartSLOduration=1.57585942 podStartE2EDuration="3.576652631s" totalImagesPullingTime="2.000793211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:18.575891926 +0000 UTC m=+224.278714123" watchObservedRunningTime="2026-09-16 00:47:18.576652631 +0000 UTC m=+224.279474820"
Sep 16 00:47:18.592135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:18.591916    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ss7zp" podStartSLOduration=3.5919017269999998 podStartE2EDuration="3.591901727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:18.5913409 +0000 UTC m=+224.294163094" watchObservedRunningTime="2026-09-16 00:47:18.591901727 +0000 UTC m=+224.294723922"
Sep 16 00:47:19.530806 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:19.530770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:19.530963 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.530926    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:47:19.531013 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.531003    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls podName:066e3b1a-2974-404d-81e0-c30a22b787ba nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.530987075 +0000 UTC m=+229.233809248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c8977" (UID: "066e3b1a-2974-404d-81e0-c30a22b787ba") : secret "samples-operator-tls" not found
Sep 16 00:47:19.632113 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:19.632079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:19.632113 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:19.632131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:19.632212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.632236    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.632244    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.632231807 +0000 UTC m=+229.335053980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : configmap references non-existent config key: service-ca.crt
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:19.632331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.632401    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.632415    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db64bf567-9gr2t: secret "image-registry-tls" not found
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.632456    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls podName:3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.632441059 +0000 UTC m=+229.335263247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls") pod "image-registry-5db64bf567-9gr2t" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e") : secret "image-registry-tls" not found
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.632485    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls podName:226d6ebd-5397-4939-b861-33f1d5da4a10 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.632465461 +0000 UTC m=+229.335287645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4r5r7" (UID: "226d6ebd-5397-4939-b861-33f1d5da4a10") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.632514    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:47:19.632651 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:19.632570    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.63255844 +0000 UTC m=+229.335380634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : secret "router-metrics-certs-default" not found
Sep 16 00:47:21.382655 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.382622    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5dcs9"]
Sep 16 00:47:21.384705 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.384683    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.386716 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.386693    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 00:47:21.386870 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.386815    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 00:47:21.386978 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.386959    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 00:47:21.387481 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.387465    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 00:47:21.387582 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.387486    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lt8cw\""
Sep 16 00:47:21.395960 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.395934    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5dcs9"]
Sep 16 00:47:21.446919 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.446893    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-crio-socket\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.447019 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.446964    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.447019 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.447004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.447146 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.447030    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszhh\" (UniqueName: \"kubernetes.io/projected/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-api-access-gszhh\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.447146 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.447141    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-data-volume\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.548224 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.548202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-data-volume\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.548343 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.548318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-crio-socket\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.548392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.548379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.548427 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.548411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.548463 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.548428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszhh\" (UniqueName: \"kubernetes.io/projected/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-api-access-gszhh\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.548463 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.548440    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-crio-socket\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.548554 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.548514    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-data-volume\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:21.548554 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:21.548546    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:21.548625 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:21.548575    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:21.548625 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:21.548600    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:22.048586467 +0000 UTC m=+227.751408640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:21.548694 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:21.548626    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:22.048610402 +0000 UTC m=+227.751432588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:21.556673 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:21.556653    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszhh\" (UniqueName: \"kubernetes.io/projected/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-api-access-gszhh\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:22.053816 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:22.053784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:22.053973 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:22.053827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:22.053973 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:22.053911    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:22.053973 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:22.053911    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:22.053973 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:22.053960    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.0539477 +0000 UTC m=+228.756769873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:22.053973 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:22.053974    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:23.053967422 +0000 UTC m=+228.756789595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:23.060888 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:23.060861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:23.061280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:23.060986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:23.061280 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.061043    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:23.061280 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.061049    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:23.061280 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.061120    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:25.061101012 +0000 UTC m=+230.763923208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:23.061280 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.061141    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:25.061131845 +0000 UTC m=+230.763954039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:23.564027 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:23.563997    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:23.564186 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.564140    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:47:23.564226 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.564200    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls podName:066e3b1a-2974-404d-81e0-c30a22b787ba nodeName:}" failed. No retries permitted until 2026-09-16 00:47:31.564182675 +0000 UTC m=+237.267004861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c8977" (UID: "066e3b1a-2974-404d-81e0-c30a22b787ba") : secret "samples-operator-tls" not found
Sep 16 00:47:23.664335 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:23.664306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:23.664451 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:23.664368    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:23.664451 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:23.664408    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:23.664545 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.664454    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:47:23.664545 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.664471    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db64bf567-9gr2t: secret "image-registry-tls" not found
Sep 16 00:47:23.664545 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.664468    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:31.664449779 +0000 UTC m=+237.367271964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : configmap references non-existent config key: service-ca.crt
Sep 16 00:47:23.664545 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.664510    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:47:23.664545 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.664534    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls podName:3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e nodeName:}" failed. No retries permitted until 2026-09-16 00:47:31.664507199 +0000 UTC m=+237.367329372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls") pod "image-registry-5db64bf567-9gr2t" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e") : secret "image-registry-tls" not found
Sep 16 00:47:23.664733 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.664569    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs podName:9ef9603f-db2f-47c9-a78a-eec9b7f52d94 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:31.664556005 +0000 UTC m=+237.367378202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs") pod "router-default-74867654d6-ccpxx" (UID: "9ef9603f-db2f-47c9-a78a-eec9b7f52d94") : secret "router-metrics-certs-default" not found
Sep 16 00:47:23.664733 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:23.664605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:23.664733 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.664699    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:23.664733 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:23.664734    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls podName:226d6ebd-5397-4939-b861-33f1d5da4a10 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:31.664723543 +0000 UTC m=+237.367545728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4r5r7" (UID: "226d6ebd-5397-4939-b861-33f1d5da4a10") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:47:24.342651 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:24.342625    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rhklt_93bd702c-41a7-4c5b-b7a2-24428b0e7a18/dns-node-resolver/0.log"
Sep 16 00:47:25.075677 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:25.075640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:25.075840 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:25.075689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:25.075840 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:25.075767    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:25.075840 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:25.075834    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:29.075817066 +0000 UTC m=+234.778639250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:25.076011 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:25.075850    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:25.076011 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:25.075895    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:29.075883299 +0000 UTC m=+234.778705475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:25.152668 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:25.152646    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ch47b_0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e/node-ca/0.log"
Sep 16 00:47:25.943174 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:25.943147    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hf8lr_e064abea-ce8c-4fad-b012-d23a5fbf95ab/serve-healthcheck-canary/0.log"
Sep 16 00:47:26.744149 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:26.744123    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9982t_99772ee8-b1af-4d1c-acd4-55ea52a1a114/kube-storage-version-migrator-operator/0.log"
Sep 16 00:47:29.107884 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:29.107851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:29.108255 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:29.107953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:29.108255 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:29.108006    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:29.108255 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:29.108011    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:47:29.108255 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:29.108065    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:37.108052934 +0000 UTC m=+242.810875107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : configmap "kube-rbac-proxy" not found
Sep 16 00:47:29.108255 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:29.108077    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls podName:66b2ae7e-9911-45c4-8247-b5b4b55d83b6 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:37.108070818 +0000 UTC m=+242.810892991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5dcs9" (UID: "66b2ae7e-9911-45c4-8247-b5b4b55d83b6") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:47:31.627315 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.627281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:31.629822 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.629804    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/066e3b1a-2974-404d-81e0-c30a22b787ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c8977\" (UID: \"066e3b1a-2974-404d-81e0-c30a22b787ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:31.728642 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.728615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:31.728749 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.728652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:31.728749 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.728700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:31.728749 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.728721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:31.729367 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.729334    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-service-ca-bundle\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:31.730990 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.730968    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/226d6ebd-5397-4939-b861-33f1d5da4a10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4r5r7\" (UID: \"226d6ebd-5397-4939-b861-33f1d5da4a10\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:31.731125 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.731105    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef9603f-db2f-47c9-a78a-eec9b7f52d94-metrics-certs\") pod \"router-default-74867654d6-ccpxx\" (UID: \"9ef9603f-db2f-47c9-a78a-eec9b7f52d94\") " pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:31.731195 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.731181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"image-registry-5db64bf567-9gr2t\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") " pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:31.750136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.750111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"
Sep 16 00:47:31.768015 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.767993    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:31.841102 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.841069    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"
Sep 16 00:47:31.849992 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:31.849973    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:32.016671 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.016642    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977"]
Sep 16 00:47:32.039923 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.039894    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74867654d6-ccpxx"]
Sep 16 00:47:32.043901 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:32.043875    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef9603f_db2f_47c9_a78a_eec9b7f52d94.slice/crio-8c38db8d33612eb206d4571569f8048f8d45a2a19a526365b133c3b4025a1b51 WatchSource:0}: Error finding container 8c38db8d33612eb206d4571569f8048f8d45a2a19a526365b133c3b4025a1b51: Status 404 returned error can't find the container with id 8c38db8d33612eb206d4571569f8048f8d45a2a19a526365b133c3b4025a1b51
Sep 16 00:47:32.109477 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.109449    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7"]
Sep 16 00:47:32.112714 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:32.112686    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226d6ebd_5397_4939_b861_33f1d5da4a10.slice/crio-4c3f5c74c637927d1f721dfa14305877b74f17afe344a078a4356a5bd5a14dd7 WatchSource:0}: Error finding container 4c3f5c74c637927d1f721dfa14305877b74f17afe344a078a4356a5bd5a14dd7: Status 404 returned error can't find the container with id 4c3f5c74c637927d1f721dfa14305877b74f17afe344a078a4356a5bd5a14dd7
Sep 16 00:47:32.142052 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.141984    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5db64bf567-9gr2t"]
Sep 16 00:47:32.145264 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:32.145237    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6b3db6_f4f1_4616_bc1c_ee9e0d43492e.slice/crio-3730e3f6ebd766e2a55430564ec4986a3e10b16d22fe7ae6f31f04e394bad22e WatchSource:0}: Error finding container 3730e3f6ebd766e2a55430564ec4986a3e10b16d22fe7ae6f31f04e394bad22e: Status 404 returned error can't find the container with id 3730e3f6ebd766e2a55430564ec4986a3e10b16d22fe7ae6f31f04e394bad22e
Sep 16 00:47:32.587992 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.587949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7" event={"ID":"226d6ebd-5397-4939-b861-33f1d5da4a10","Type":"ContainerStarted","Data":"4c3f5c74c637927d1f721dfa14305877b74f17afe344a078a4356a5bd5a14dd7"}
Sep 16 00:47:32.589355 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.589329    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t" event={"ID":"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e","Type":"ContainerStarted","Data":"febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c"}
Sep 16 00:47:32.589478 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.589375    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t" event={"ID":"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e","Type":"ContainerStarted","Data":"3730e3f6ebd766e2a55430564ec4986a3e10b16d22fe7ae6f31f04e394bad22e"}
Sep 16 00:47:32.589478 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.589401    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:47:32.590404 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.590382    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977" event={"ID":"066e3b1a-2974-404d-81e0-c30a22b787ba","Type":"ContainerStarted","Data":"ed33a22daac7c7b5f4e18b184c419b6038d7365e71c3b605c0fbdc9a47788899"}
Sep 16 00:47:32.591772 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.591747    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74867654d6-ccpxx" event={"ID":"9ef9603f-db2f-47c9-a78a-eec9b7f52d94","Type":"ContainerStarted","Data":"b3931742cdc53ee6d72526e2d198173d2e1ba0732c2205ec58cb6e6fa698f651"}
Sep 16 00:47:32.591878 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.591781    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74867654d6-ccpxx" event={"ID":"9ef9603f-db2f-47c9-a78a-eec9b7f52d94","Type":"ContainerStarted","Data":"8c38db8d33612eb206d4571569f8048f8d45a2a19a526365b133c3b4025a1b51"}
Sep 16 00:47:32.606710 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.606666    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t" podStartSLOduration=17.606652738 podStartE2EDuration="17.606652738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:32.60575086 +0000 UTC m=+238.308573056" watchObservedRunningTime="2026-09-16 00:47:32.606652738 +0000 UTC m=+238.309474930"
Sep 16 00:47:32.622421 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.622379    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-74867654d6-ccpxx" podStartSLOduration=17.622365517 podStartE2EDuration="17.622365517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:32.621740919 +0000 UTC m=+238.324563137" watchObservedRunningTime="2026-09-16 00:47:32.622365517 +0000 UTC m=+238.325187712"
Sep 16 00:47:32.769606 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.769563    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:32.770887 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:32.770864    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:33.595022 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:33.594986    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:33.596346 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:33.596325    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74867654d6-ccpxx"
Sep 16 00:47:34.599975 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:34.599894    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7" event={"ID":"226d6ebd-5397-4939-b861-33f1d5da4a10","Type":"ContainerStarted","Data":"1eb1a682761f828d107b1c354d20d7cbe3bf86e4ca31f000c0b119a0dce65788"}
Sep 16 00:47:34.601690 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:34.601665    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977" event={"ID":"066e3b1a-2974-404d-81e0-c30a22b787ba","Type":"ContainerStarted","Data":"86833d9c6bc90aec6c3e1642768b5c5446ea724d9eecf1042008589c1fc81292"}
Sep 16 00:47:34.601822 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:34.601706    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977" event={"ID":"066e3b1a-2974-404d-81e0-c30a22b787ba","Type":"ContainerStarted","Data":"b008dcffacd263e4cee394e5048a044b0dd543e62641280c7d782da87edbf80d"}
Sep 16 00:47:34.614856 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:34.614803    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4r5r7" podStartSLOduration=17.495921056 podStartE2EDuration="19.614788705s" totalImagesPullingTime="2.118867649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:34.614072625 +0000 UTC m=+240.316894821" watchObservedRunningTime="2026-09-16 00:47:34.614788705 +0000 UTC m=+240.317610896"
Sep 16 00:47:34.629760 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:34.629720    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c8977" podStartSLOduration=17.453523976 podStartE2EDuration="19.62970488s" totalImagesPullingTime="2.176180904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:34.629251163 +0000 UTC m=+240.332073355" watchObservedRunningTime="2026-09-16 00:47:34.62970488 +0000 UTC m=+240.332527079"
Sep 16 00:47:37.175682 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:37.175633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:37.176074 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:37.175723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:37.176168 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:37.176152    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:37.178131 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:37.178096    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66b2ae7e-9911-45c4-8247-b5b4b55d83b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5dcs9\" (UID: \"66b2ae7e-9911-45c4-8247-b5b4b55d83b6\") " pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:37.296544 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:37.296496    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lt8cw\""
Sep 16 00:47:37.305219 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:37.305198    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5dcs9"
Sep 16 00:47:37.574458 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:37.574423    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5dcs9"]
Sep 16 00:47:37.577314 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:37.577288    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b2ae7e_9911_45c4_8247_b5b4b55d83b6.slice/crio-d5dc94d0256add30b80287589146280e576ce740b68f41730612cc617916a29a WatchSource:0}: Error finding container d5dc94d0256add30b80287589146280e576ce740b68f41730612cc617916a29a: Status 404 returned error can't find the container with id d5dc94d0256add30b80287589146280e576ce740b68f41730612cc617916a29a
Sep 16 00:47:37.612456 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:37.612427    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5dcs9" event={"ID":"66b2ae7e-9911-45c4-8247-b5b4b55d83b6","Type":"ContainerStarted","Data":"d5dc94d0256add30b80287589146280e576ce740b68f41730612cc617916a29a"}
Sep 16 00:47:38.617649 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:38.617568    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5dcs9" event={"ID":"66b2ae7e-9911-45c4-8247-b5b4b55d83b6","Type":"ContainerStarted","Data":"93e248a79af1f998bd5caf28932380ecfdbf7c038f5e6ffcdf4c148d40d62650"}
Sep 16 00:47:38.617649 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:38.617620    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5dcs9" event={"ID":"66b2ae7e-9911-45c4-8247-b5b4b55d83b6","Type":"ContainerStarted","Data":"9df1ca5d366ce028210971378105b55916e4743b28448819b8a40d1a94de4e50"}
Sep 16 00:47:40.626254 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:40.626222    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5dcs9" event={"ID":"66b2ae7e-9911-45c4-8247-b5b4b55d83b6","Type":"ContainerStarted","Data":"bc18f84a7505611eea9fc6c42c8759f130626ca75b86b06c60a6df1e182514a1"}
Sep 16 00:47:40.643246 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:40.643198    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5dcs9" podStartSLOduration=17.660614837 podStartE2EDuration="19.643183043s" totalImagesPullingTime="1.982568206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:47:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:40.641938315 +0000 UTC m=+246.344760522" watchObservedRunningTime="2026-09-16 00:47:40.643183043 +0000 UTC m=+246.346005255"
Sep 16 00:47:42.263263 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.263234    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"]
Sep 16 00:47:42.266361 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.266344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"
Sep 16 00:47:42.270181 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.270163    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-c54bw\""
Sep 16 00:47:42.270291 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.270199    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 00:47:42.284040 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.279096    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"]
Sep 16 00:47:42.291762 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.291735    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5db64bf567-9gr2t"]
Sep 16 00:47:42.310455 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.310425    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/49971735-36c4-47ad-b8f9-2afd41dd2015-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cn7nr\" (UID: \"49971735-36c4-47ad-b8f9-2afd41dd2015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"
Sep 16 00:47:42.317807 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.317781    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68596fcf69-rkdfq"]
Sep 16 00:47:42.320699 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.320685    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.334123 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.334103    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68596fcf69-rkdfq"]
Sep 16 00:47:42.411248 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411219    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9bc456b8-82e4-49ce-9b21-bd54571af7dd-installation-pull-secrets\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.411424 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-registry-tls\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.411424 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411297    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9bc456b8-82e4-49ce-9b21-bd54571af7dd-image-registry-private-configuration\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.411424 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411331    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9bc456b8-82e4-49ce-9b21-bd54571af7dd-ca-trust-extracted\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.411424 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jds\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-kube-api-access-n6jds\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.411665 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411433    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-bound-sa-token\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.411665 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411460    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bc456b8-82e4-49ce-9b21-bd54571af7dd-trusted-ca\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.411665 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/49971735-36c4-47ad-b8f9-2afd41dd2015-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cn7nr\" (UID: \"49971735-36c4-47ad-b8f9-2afd41dd2015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"
Sep 16 00:47:42.411665 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.411566    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9bc456b8-82e4-49ce-9b21-bd54571af7dd-registry-certificates\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.413979 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.413951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/49971735-36c4-47ad-b8f9-2afd41dd2015-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cn7nr\" (UID: \"49971735-36c4-47ad-b8f9-2afd41dd2015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"
Sep 16 00:47:42.511972 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.511900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9bc456b8-82e4-49ce-9b21-bd54571af7dd-registry-certificates\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.511972 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.511959    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9bc456b8-82e4-49ce-9b21-bd54571af7dd-installation-pull-secrets\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.512142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.511994    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-registry-tls\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.512142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.512026    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9bc456b8-82e4-49ce-9b21-bd54571af7dd-image-registry-private-configuration\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.512142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.512060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9bc456b8-82e4-49ce-9b21-bd54571af7dd-ca-trust-extracted\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.512142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.512087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jds\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-kube-api-access-n6jds\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.512142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.512129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-bound-sa-token\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.512369 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.512159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bc456b8-82e4-49ce-9b21-bd54571af7dd-trusted-ca\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.512587 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.512555    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9bc456b8-82e4-49ce-9b21-bd54571af7dd-ca-trust-extracted\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.512957 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.512933    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9bc456b8-82e4-49ce-9b21-bd54571af7dd-registry-certificates\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.513166 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.513149    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bc456b8-82e4-49ce-9b21-bd54571af7dd-trusted-ca\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.514657 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.514628    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9bc456b8-82e4-49ce-9b21-bd54571af7dd-installation-pull-secrets\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.514822 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.514802    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9bc456b8-82e4-49ce-9b21-bd54571af7dd-image-registry-private-configuration\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.514880 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.514867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-registry-tls\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.519774 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.519752    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-bound-sa-token\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.520178 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.520158    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jds\" (UniqueName: \"kubernetes.io/projected/9bc456b8-82e4-49ce-9b21-bd54571af7dd-kube-api-access-n6jds\") pod \"image-registry-68596fcf69-rkdfq\" (UID: \"9bc456b8-82e4-49ce-9b21-bd54571af7dd\") " pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.575253 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.575216    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"
Sep 16 00:47:42.628838 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.628815    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:42.844815 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.844748    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"]
Sep 16 00:47:42.847933 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:42.847912    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49971735_36c4_47ad_b8f9_2afd41dd2015.slice/crio-31fd6f2dda392dc636e78fa6742df51421a658ea64038b3e429dfd1e9d18ae96 WatchSource:0}: Error finding container 31fd6f2dda392dc636e78fa6742df51421a658ea64038b3e429dfd1e9d18ae96: Status 404 returned error can't find the container with id 31fd6f2dda392dc636e78fa6742df51421a658ea64038b3e429dfd1e9d18ae96
Sep 16 00:47:42.901912 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:42.901883    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68596fcf69-rkdfq"]
Sep 16 00:47:42.904592 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:42.904568    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc456b8_82e4_49ce_9b21_bd54571af7dd.slice/crio-d630d8f6a33e794c430f24c84c3d38fea4eac1b2b0286732d9374855fbe66b28 WatchSource:0}: Error finding container d630d8f6a33e794c430f24c84c3d38fea4eac1b2b0286732d9374855fbe66b28: Status 404 returned error can't find the container with id d630d8f6a33e794c430f24c84c3d38fea4eac1b2b0286732d9374855fbe66b28
Sep 16 00:47:43.637595 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:43.637562    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68596fcf69-rkdfq" event={"ID":"9bc456b8-82e4-49ce-9b21-bd54571af7dd","Type":"ContainerStarted","Data":"75a901c931020240b581b12a16b86648288c72a95bb0bbff9677b5ea092da28a"}
Sep 16 00:47:43.638109 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:43.637622    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68596fcf69-rkdfq" event={"ID":"9bc456b8-82e4-49ce-9b21-bd54571af7dd","Type":"ContainerStarted","Data":"d630d8f6a33e794c430f24c84c3d38fea4eac1b2b0286732d9374855fbe66b28"}
Sep 16 00:47:43.638109 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:43.637650    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:47:43.639322 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:43.639297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr" event={"ID":"49971735-36c4-47ad-b8f9-2afd41dd2015","Type":"ContainerStarted","Data":"31fd6f2dda392dc636e78fa6742df51421a658ea64038b3e429dfd1e9d18ae96"}
Sep 16 00:47:43.654863 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:43.654824    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68596fcf69-rkdfq" podStartSLOduration=1.654809212 podStartE2EDuration="1.654809212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:43.653971529 +0000 UTC m=+249.356793725" watchObservedRunningTime="2026-09-16 00:47:43.654809212 +0000 UTC m=+249.357631407"
Sep 16 00:47:44.643862 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:44.643823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr" event={"ID":"49971735-36c4-47ad-b8f9-2afd41dd2015","Type":"ContainerStarted","Data":"bd532370d32fcacbd6351315848628de475c4f008213a1fc06eec992dcae434e"}
Sep 16 00:47:44.644317 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:44.644049    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"
Sep 16 00:47:44.648503 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:44.648486    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr"
Sep 16 00:47:44.661005 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:44.660953    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cn7nr" podStartSLOduration=1.570036005 podStartE2EDuration="2.660938959s" totalImagesPullingTime="1.090902954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:44.659873467 +0000 UTC m=+250.362695663" watchObservedRunningTime="2026-09-16 00:47:44.660938959 +0000 UTC m=+250.363761154"
Sep 16 00:47:45.635904 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:45.635865    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:47:45.638288 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:45.638267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5527c19f-cd6a-4cd6-adc4-0bb7e70f140d-metrics-certs\") pod \"network-metrics-daemon-nbb66\" (UID: \"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d\") " pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:47:45.868790 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:45.868759    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6btht\""
Sep 16 00:47:45.876281 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:45.876261    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbb66"
Sep 16 00:47:46.144713 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:46.144683    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nbb66"]
Sep 16 00:47:46.148038 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:46.148015    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5527c19f_cd6a_4cd6_adc4_0bb7e70f140d.slice/crio-6e5e2e396608cb0824d73c2770d51a248cab4708d282d8225ee00fb44140820e WatchSource:0}: Error finding container 6e5e2e396608cb0824d73c2770d51a248cab4708d282d8225ee00fb44140820e: Status 404 returned error can't find the container with id 6e5e2e396608cb0824d73c2770d51a248cab4708d282d8225ee00fb44140820e
Sep 16 00:47:46.651409 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:46.651374    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nbb66" event={"ID":"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d","Type":"ContainerStarted","Data":"6e5e2e396608cb0824d73c2770d51a248cab4708d282d8225ee00fb44140820e"}
Sep 16 00:47:47.656426 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:47.656342    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nbb66" event={"ID":"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d","Type":"ContainerStarted","Data":"83ffc5ec242ae6962e37d0361c8f6332d95223b58c0954461d3e6354eaa3a200"}
Sep 16 00:47:47.656426 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:47.656400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nbb66" event={"ID":"5527c19f-cd6a-4cd6-adc4-0bb7e70f140d","Type":"ContainerStarted","Data":"3b072c4fe18c9fc2398112893e8106cff72d2959f063192f17a57c9b938e656e"}
Sep 16 00:47:47.670652 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:47.670602    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nbb66" podStartSLOduration=250.539138484 podStartE2EDuration="4m11.670585701s" totalImagesPullingTime="1.131447217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:47.670029954 +0000 UTC m=+253.372852162" watchObservedRunningTime="2026-09-16 00:47:47.670585701 +0000 UTC m=+253.373407895"
Sep 16 00:47:49.201803 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.201769    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s6jj7"]
Sep 16 00:47:49.206682 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.206665    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.208607 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.208588    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 00:47:49.208952 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.208937    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ldbkq\""
Sep 16 00:47:49.209032 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.208982    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 00:47:49.209111 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.209040    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 00:47:49.209371 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.209351    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 00:47:49.266480 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-tls\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.266598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-textfile\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.266598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c884b58b-cf9d-43d6-a79d-2f25cbebd729-metrics-client-ca\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.266598 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266559    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-root\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.266733 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266617    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vcd\" (UniqueName: \"kubernetes.io/projected/c884b58b-cf9d-43d6-a79d-2f25cbebd729-kube-api-access-s4vcd\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.266733 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266646    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-sys\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.266733 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266675    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.266733 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.266864 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.266753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-wtmp\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368012 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.367972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-root\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368012 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368016    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vcd\" (UniqueName: \"kubernetes.io/projected/c884b58b-cf9d-43d6-a79d-2f25cbebd729-kube-api-access-s4vcd\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368034    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-sys\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368105    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-root\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368150    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368180    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-sys\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-wtmp\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368274    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-tls\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.368348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-textfile\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.369185 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c884b58b-cf9d-43d6-a79d-2f25cbebd729-metrics-client-ca\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.369185 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368387    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-wtmp\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.369185 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:49.368496    2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 00:47:49.369185 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:47:49.368591    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-tls podName:c884b58b-cf9d-43d6-a79d-2f25cbebd729 nodeName:}" failed. No retries permitted until 2026-09-16 00:47:49.868570632 +0000 UTC m=+255.571392809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-tls") pod "node-exporter-s6jj7" (UID: "c884b58b-cf9d-43d6-a79d-2f25cbebd729") : secret "node-exporter-tls" not found
Sep 16 00:47:49.369185 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-textfile\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.369185 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368932    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.369185 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.368983    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c884b58b-cf9d-43d6-a79d-2f25cbebd729-metrics-client-ca\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.370993 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.370972    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.376212 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.376187    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vcd\" (UniqueName: \"kubernetes.io/projected/c884b58b-cf9d-43d6-a79d-2f25cbebd729-kube-api-access-s4vcd\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.871449 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.871414    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-tls\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:49.873753 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:49.873736    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c884b58b-cf9d-43d6-a79d-2f25cbebd729-node-exporter-tls\") pod \"node-exporter-s6jj7\" (UID: \"c884b58b-cf9d-43d6-a79d-2f25cbebd729\") " pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:50.115707 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:50.115675    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6jj7"
Sep 16 00:47:50.127326 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:50.127303    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc884b58b_cf9d_43d6_a79d_2f25cbebd729.slice/crio-02c4b0c5a3942aae5a48d892f01c02e828d893ac5783024e8da37de0434f74d3 WatchSource:0}: Error finding container 02c4b0c5a3942aae5a48d892f01c02e828d893ac5783024e8da37de0434f74d3: Status 404 returned error can't find the container with id 02c4b0c5a3942aae5a48d892f01c02e828d893ac5783024e8da37de0434f74d3
Sep 16 00:47:50.666650 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:50.666610    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6jj7" event={"ID":"c884b58b-cf9d-43d6-a79d-2f25cbebd729","Type":"ContainerStarted","Data":"02c4b0c5a3942aae5a48d892f01c02e828d893ac5783024e8da37de0434f74d3"}
Sep 16 00:47:51.672175 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:51.672141    2573 generic.go:356] "Generic (PLEG): container finished" podID="c884b58b-cf9d-43d6-a79d-2f25cbebd729" containerID="43f8a11fa45cd3240e95afbd7ba95f101c37a1c08d63f4f463f81d05cc4549af" exitCode=0
Sep 16 00:47:51.672639 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:51.672183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6jj7" event={"ID":"c884b58b-cf9d-43d6-a79d-2f25cbebd729","Type":"ContainerDied","Data":"43f8a11fa45cd3240e95afbd7ba95f101c37a1c08d63f4f463f81d05cc4549af"}
Sep 16 00:47:52.267218 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.267189    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78b986c49b-ptn4w"]
Sep 16 00:47:52.270759 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.270737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.272891 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.272871    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 00:47:52.273041 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.272957    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 00:47:52.273110 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.273097    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6gxgr\""
Sep 16 00:47:52.273165 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.273100    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 00:47:52.273220 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.273100    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 00:47:52.273220 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.273104    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 00:47:52.273316 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.273243    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5l33ks5lal8t\""
Sep 16 00:47:52.283437 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.283416    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78b986c49b-ptn4w"]
Sep 16 00:47:52.289287 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.289263    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-tls\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.289394 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.289297    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.289394 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.289313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.289394 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.289330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.289394 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.289386    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.289650 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.289413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-grpc-tls\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.289650 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.289470    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75ee21fc-48e2-48c7-a47d-6fafbe636418-metrics-client-ca\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.289650 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.289494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2h9\" (UniqueName: \"kubernetes.io/projected/75ee21fc-48e2-48c7-a47d-6fafbe636418-kube-api-access-4r2h9\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.297534 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.297502    2573 patch_prober.go:28] interesting pod/image-registry-5db64bf567-9gr2t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 00:47:52.297618 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.297566    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t" podUID="3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:47:52.390334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.390306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.390445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.390350    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.390445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.390392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-grpc-tls\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.390445 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.390434    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75ee21fc-48e2-48c7-a47d-6fafbe636418-metrics-client-ca\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.390640 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.390459    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2h9\" (UniqueName: \"kubernetes.io/projected/75ee21fc-48e2-48c7-a47d-6fafbe636418-kube-api-access-4r2h9\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.390640 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.390503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-tls\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.390640 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.390578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.390640 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.390604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.391240 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.391215    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75ee21fc-48e2-48c7-a47d-6fafbe636418-metrics-client-ca\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.393238 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.393216    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.393330 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.393311    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.393594 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.393577    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.393825 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.393808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-grpc-tls\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.393864 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.393842    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.393901 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.393859    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/75ee21fc-48e2-48c7-a47d-6fafbe636418-secret-thanos-querier-tls\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.397884 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.397863    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2h9\" (UniqueName: \"kubernetes.io/projected/75ee21fc-48e2-48c7-a47d-6fafbe636418-kube-api-access-4r2h9\") pod \"thanos-querier-78b986c49b-ptn4w\" (UID: \"75ee21fc-48e2-48c7-a47d-6fafbe636418\") " pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.580058 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.580010    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:52.677938 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.677900    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6jj7" event={"ID":"c884b58b-cf9d-43d6-a79d-2f25cbebd729","Type":"ContainerStarted","Data":"7d83a7c755f44a9cf0de7e4ff3e8e5e49529b48e7ba0cb64030242103a747c42"}
Sep 16 00:47:52.678279 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.677966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6jj7" event={"ID":"c884b58b-cf9d-43d6-a79d-2f25cbebd729","Type":"ContainerStarted","Data":"645dcccc2f5c40dd2da68a5dadfb552ed5205934996024170a29b8ab7d0f4a54"}
Sep 16 00:47:52.695069 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.695023    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s6jj7" podStartSLOduration=2.57597362 podStartE2EDuration="3.695006814s" totalImagesPullingTime="1.049745657s" totalInitContainerRuntime="69.287537ms" isStatefulPod=true podCreationTimestamp="2026-09-16 00:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:52.694407604 +0000 UTC m=+258.397229804" watchObservedRunningTime="2026-09-16 00:47:52.695006814 +0000 UTC m=+258.397829008"
Sep 16 00:47:52.859780 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:52.859705    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78b986c49b-ptn4w"]
Sep 16 00:47:52.863195 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:52.863159    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ee21fc_48e2_48c7_a47d_6fafbe636418.slice/crio-b7674da29fefe263eeb2092d31eb1fe081365c301145e17b9b53e933160e3eed WatchSource:0}: Error finding container b7674da29fefe263eeb2092d31eb1fe081365c301145e17b9b53e933160e3eed: Status 404 returned error can't find the container with id b7674da29fefe263eeb2092d31eb1fe081365c301145e17b9b53e933160e3eed
Sep 16 00:47:53.610647 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.610617    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6696656695-2x5tk"]
Sep 16 00:47:53.614585 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.614565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.618046 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.618011    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 00:47:53.618046 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.618024    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 00:47:53.618046 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.618020    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 00:47:53.618274 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.618091    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-c5fqn\""
Sep 16 00:47:53.618274 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.618034    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 00:47:53.618274 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.618015    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ejpvjo0ak7cth\""
Sep 16 00:47:53.622646 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.622596    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6696656695-2x5tk"]
Sep 16 00:47:53.683500 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.683459    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w" event={"ID":"75ee21fc-48e2-48c7-a47d-6fafbe636418","Type":"ContainerStarted","Data":"b7674da29fefe263eeb2092d31eb1fe081365c301145e17b9b53e933160e3eed"}
Sep 16 00:47:53.700125 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.700099    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78375bf6-9446-43bd-8027-77162a7315d0-audit-log\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.700263 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.700140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78375bf6-9446-43bd-8027-77162a7315d0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.700263 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.700175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-588lx\" (UniqueName: \"kubernetes.io/projected/78375bf6-9446-43bd-8027-77162a7315d0-kube-api-access-588lx\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.700263 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.700244    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78375bf6-9446-43bd-8027-77162a7315d0-metrics-server-audit-profiles\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.700419 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.700266    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-client-ca-bundle\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.700419 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.700327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-secret-metrics-server-tls\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.700419 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.700356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-secret-metrics-server-client-certs\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.801115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.801073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78375bf6-9446-43bd-8027-77162a7315d0-audit-log\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.801250 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.801136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78375bf6-9446-43bd-8027-77162a7315d0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.801250 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.801212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-588lx\" (UniqueName: \"kubernetes.io/projected/78375bf6-9446-43bd-8027-77162a7315d0-kube-api-access-588lx\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.801250 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.801243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78375bf6-9446-43bd-8027-77162a7315d0-metrics-server-audit-profiles\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.801473 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.801270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-client-ca-bundle\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.801473 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.801367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-secret-metrics-server-tls\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.801473 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.801416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-secret-metrics-server-client-certs\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.801659 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.801498    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78375bf6-9446-43bd-8027-77162a7315d0-audit-log\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.802265 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.802215    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78375bf6-9446-43bd-8027-77162a7315d0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.802538 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.802494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78375bf6-9446-43bd-8027-77162a7315d0-metrics-server-audit-profiles\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.804750 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.804713    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-client-ca-bundle\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.804860 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.804810    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-secret-metrics-server-client-certs\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.804947 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.804922    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78375bf6-9446-43bd-8027-77162a7315d0-secret-metrics-server-tls\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.809959 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.809937    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-588lx\" (UniqueName: \"kubernetes.io/projected/78375bf6-9446-43bd-8027-77162a7315d0-kube-api-access-588lx\") pod \"metrics-server-6696656695-2x5tk\" (UID: \"78375bf6-9446-43bd-8027-77162a7315d0\") " pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:53.928149 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:53.928080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:47:54.212113 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:54.212087    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6696656695-2x5tk"]
Sep 16 00:47:54.639201 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:54.639122    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78375bf6_9446_43bd_8027_77162a7315d0.slice/crio-2a388e9c699c0f6aee3eff593192e429828346ad542889903e4d7d3a776f66a9 WatchSource:0}: Error finding container 2a388e9c699c0f6aee3eff593192e429828346ad542889903e4d7d3a776f66a9: Status 404 returned error can't find the container with id 2a388e9c699c0f6aee3eff593192e429828346ad542889903e4d7d3a776f66a9
Sep 16 00:47:54.687887 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:54.687862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6696656695-2x5tk" event={"ID":"78375bf6-9446-43bd-8027-77162a7315d0","Type":"ContainerStarted","Data":"2a388e9c699c0f6aee3eff593192e429828346ad542889903e4d7d3a776f66a9"}
Sep 16 00:47:55.412296 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.412264    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:47:55.416605 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.416581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.419734 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.419708    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 00:47:55.419846 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.419817    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fcqh6uqer9qrk\""
Sep 16 00:47:55.420255 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.420180    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 00:47:55.421391 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.420782    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 00:47:55.421391 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.420805    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 00:47:55.421391 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421087    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 00:47:55.421391 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421253    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 00:47:55.421391 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421255    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 00:47:55.421705 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421474    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 00:47:55.421705 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421502    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 00:47:55.421705 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421553    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 00:47:55.421939 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421888    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 00:47:55.421939 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421893    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 00:47:55.422051 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.421955    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 00:47:55.422184 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.422166    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h7t6g\""
Sep 16 00:47:55.423760 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.423742    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 00:47:55.438755 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.438726    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.514778    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.514819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.514846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.514892    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.514945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.514969    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515003    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config-out\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515064 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515206    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515235    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-web-config\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515374    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.515643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.515426    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4dx\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-kube-api-access-nt4dx\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616754 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616754 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616757    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616984 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616984 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616984 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616984 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config-out\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616984 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616984 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616897    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616984 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616938    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.616984 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.616965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617008    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617095    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-web-config\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617120    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617225    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4dx\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-kube-api-access-nt4dx\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.617364 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.618036 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617561    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.618036 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.618036 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.617939    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.620186 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.618799    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.620186 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.619393    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.620186 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.619881    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.621083 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.620591    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.621083 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.620688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.621083 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.620950    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.621083 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.621029    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.621355 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.621341    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.621588 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.621549    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config-out\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.621812 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.621785    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.622280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.622256    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-web-config\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.623144 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.623115    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.623325 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.623287    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.623430 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.623294    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.624036 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.624013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.624796 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.624776    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.627717 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.627694    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4dx\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-kube-api-access-nt4dx\") pod \"prometheus-k8s-0\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:55.694139 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.694065    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w" event={"ID":"75ee21fc-48e2-48c7-a47d-6fafbe636418","Type":"ContainerStarted","Data":"0b4d0ed24fc9557575ef0de8c7fefa77fa41505fccc18fa5cecd6aad97baf8e7"}
Sep 16 00:47:55.694496 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.694179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w" event={"ID":"75ee21fc-48e2-48c7-a47d-6fafbe636418","Type":"ContainerStarted","Data":"0b81d826fa93c49f571233c576fe47c921f436c43866f91b44138e5f12014255"}
Sep 16 00:47:55.694496 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.694226    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w" event={"ID":"75ee21fc-48e2-48c7-a47d-6fafbe636418","Type":"ContainerStarted","Data":"f7feccddcd2f28343897ef103151c57b94f1ba7122d6624842957c2192dfd838"}
Sep 16 00:47:55.729474 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:55.729450    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:47:56.369017 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.368985    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:47:56.370705 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:47:56.370682    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dbddd5_e888_4fc5_bb9d_1a2d04ecd446.slice/crio-0e9456fea5f660fdf55c9235cece525c0352545195cae4c485e055792711d13f WatchSource:0}: Error finding container 0e9456fea5f660fdf55c9235cece525c0352545195cae4c485e055792711d13f: Status 404 returned error can't find the container with id 0e9456fea5f660fdf55c9235cece525c0352545195cae4c485e055792711d13f
Sep 16 00:47:56.702083 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.702053    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w" event={"ID":"75ee21fc-48e2-48c7-a47d-6fafbe636418","Type":"ContainerStarted","Data":"5b4c21949e5e90387bc9694c6fd7988d4a45be8acec61df588266b16304b5a94"}
Sep 16 00:47:56.702548 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.702128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w" event={"ID":"75ee21fc-48e2-48c7-a47d-6fafbe636418","Type":"ContainerStarted","Data":"ffe5532173ea08a70ed7b19e4038c6a9fa8272754f6a003d7b8f62650b50e531"}
Sep 16 00:47:56.702548 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.702181    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:47:56.702548 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.702220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w" event={"ID":"75ee21fc-48e2-48c7-a47d-6fafbe636418","Type":"ContainerStarted","Data":"669550b42de8dfd74580bf5f369e3dd0851a516cee925438046fb5820a1de526"}
Sep 16 00:47:56.703326 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.703301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerStarted","Data":"0e9456fea5f660fdf55c9235cece525c0352545195cae4c485e055792711d13f"}
Sep 16 00:47:56.704763 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.704739    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6696656695-2x5tk" event={"ID":"78375bf6-9446-43bd-8027-77162a7315d0","Type":"ContainerStarted","Data":"718399e0427f7724d627f0e156428915674fd4bf0c05afd3815e4aac48cd1926"}
Sep 16 00:47:56.729291 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.729245    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w" podStartSLOduration=1.677919629 podStartE2EDuration="4.729226764s" totalImagesPullingTime="3.051307135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:56.727801057 +0000 UTC m=+262.430623254" watchObservedRunningTime="2026-09-16 00:47:56.729226764 +0000 UTC m=+262.432048960"
Sep 16 00:47:56.743662 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:56.743619    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6696656695-2x5tk" podStartSLOduration=2.2940359790000002 podStartE2EDuration="3.743603548s" totalImagesPullingTime="1.449567569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:56.742405409 +0000 UTC m=+262.445227606" watchObservedRunningTime="2026-09-16 00:47:56.743603548 +0000 UTC m=+262.446425744"
Sep 16 00:47:57.710777 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:57.710746    2573 generic.go:356] "Generic (PLEG): container finished" podID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerID="06c9e9f0e8963ce9bb793feba5de5cf617816399c5929076b58fa5cd20dc4655" exitCode=0
Sep 16 00:47:57.711212 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:47:57.710835    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerDied","Data":"06c9e9f0e8963ce9bb793feba5de5cf617816399c5929076b58fa5cd20dc4655"}
Sep 16 00:48:01.731413 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:01.731381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerStarted","Data":"5c6dba5b859c87f84e8082851d619c7850bdccc9eac4a5b24cdfb56fe34eafca"}
Sep 16 00:48:01.731780 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:01.731443    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerStarted","Data":"3cfaed5200a817f8a65c343352c0f9d0981112f3c663eb6c3b66490ce8ec00fe"}
Sep 16 00:48:01.731780 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:01.731469    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerStarted","Data":"b1c44f780c48887ebfd35a2bb68544db34990f1b5725d29bc6ce757fdb66a650"}
Sep 16 00:48:01.731780 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:01.731495    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerStarted","Data":"3193c1e3be7bf96d87726e35cf297025f3c7c25501238870d7c6cafd36221628"}
Sep 16 00:48:01.731780 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:01.731538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerStarted","Data":"26fd191bd5cdcb3ebc659a28f540a91839869aa0519e3495f740ab5723d14ca0"}
Sep 16 00:48:01.731780 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:01.731561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerStarted","Data":"d426c0d49680eb8b8ddb97729dbcbd31ef5a8bb88e818c99fdd9046d8b2953a6"}
Sep 16 00:48:01.758556 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:01.758492    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.427123055 podStartE2EDuration="6.758475026s" totalImagesPullingTime="4.311746623s" totalInitContainerRuntime="19.605348ms" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:01.755797894 +0000 UTC m=+267.458620089" watchObservedRunningTime="2026-09-16 00:48:01.758475026 +0000 UTC m=+267.461297250"
Sep 16 00:48:02.296546 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:02.296491    2573 patch_prober.go:28] interesting pod/image-registry-5db64bf567-9gr2t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 00:48:02.296724 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:02.296582    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t" podUID="3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:48:02.633863 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:02.633776    2573 patch_prober.go:28] interesting pod/image-registry-68596fcf69-rkdfq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 00:48:02.633863 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:02.633827    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-68596fcf69-rkdfq" podUID="9bc456b8-82e4-49ce-9b21-bd54571af7dd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:48:02.722081 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:02.722050    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78b986c49b-ptn4w"
Sep 16 00:48:04.650296 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:04.650265    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68596fcf69-rkdfq"
Sep 16 00:48:05.730192 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:05.730154    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:48:07.310596 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.310492    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t" podUID="3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" containerName="registry" containerID="cri-o://febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c" gracePeriod=30
Sep 16 00:48:07.557359 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.557337    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:48:07.627280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627186    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-trusted-ca\") pod \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") "
Sep 16 00:48:07.627280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627247    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsgsj\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-kube-api-access-vsgsj\") pod \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") "
Sep 16 00:48:07.627280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627275    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-ca-trust-extracted\") pod \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") "
Sep 16 00:48:07.627584 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627298    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-installation-pull-secrets\") pod \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") "
Sep 16 00:48:07.627584 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627313    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-certificates\") pod \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") "
Sep 16 00:48:07.627584 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627342    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-image-registry-private-configuration\") pod \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") "
Sep 16 00:48:07.627584 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627371    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") pod \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") "
Sep 16 00:48:07.627584 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627398    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-bound-sa-token\") pod \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\" (UID: \"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e\") "
Sep 16 00:48:07.627843 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627717    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:48:07.627843 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.627829    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:48:07.630147 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.630108    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-kube-api-access-vsgsj" (OuterVolumeSpecName: "kube-api-access-vsgsj") pod "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e"). InnerVolumeSpecName "kube-api-access-vsgsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:48:07.630147 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.630139    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:48:07.630303 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.630187    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:48:07.630390 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.630363    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:48:07.630471 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.630412    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:48:07.635809 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.635788    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" (UID: "3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:48:07.728841 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.728808    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-bound-sa-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:48:07.728841 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.728833    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-trusted-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:48:07.728841 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.728841    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsgsj\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-kube-api-access-vsgsj\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:48:07.729058 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.728851    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-ca-trust-extracted\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:48:07.729058 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.728860    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-installation-pull-secrets\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:48:07.729058 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.728868    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-certificates\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:48:07.729058 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.728877    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-image-registry-private-configuration\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:48:07.729058 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.728886    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e-registry-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:48:07.757371 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.757335    2573 generic.go:356] "Generic (PLEG): container finished" podID="3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" containerID="febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c" exitCode=0
Sep 16 00:48:07.757501 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.757393    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t"
Sep 16 00:48:07.757501 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.757406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t" event={"ID":"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e","Type":"ContainerDied","Data":"febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c"}
Sep 16 00:48:07.757501 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.757448    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db64bf567-9gr2t" event={"ID":"3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e","Type":"ContainerDied","Data":"3730e3f6ebd766e2a55430564ec4986a3e10b16d22fe7ae6f31f04e394bad22e"}
Sep 16 00:48:07.757501 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.757480    2573 scope.go:117] "RemoveContainer" containerID="febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c"
Sep 16 00:48:07.766511 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.766489    2573 scope.go:117] "RemoveContainer" containerID="febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c"
Sep 16 00:48:07.766793 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:48:07.766772    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c\": container with ID starting with febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c not found: ID does not exist" containerID="febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c"
Sep 16 00:48:07.766859 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.766801    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c"} err="failed to get container status \"febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c\": rpc error: code = NotFound desc = could not find container \"febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c\": container with ID starting with febd9ea7bae164bdc96c510ee37889bd4864bb69d1a528779e3925187df4614c not found: ID does not exist"
Sep 16 00:48:07.777181 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.777160    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5db64bf567-9gr2t"]
Sep 16 00:48:07.780974 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:07.780942    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5db64bf567-9gr2t"]
Sep 16 00:48:08.769013 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:08.768984    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" path="/var/lib/kubelet/pods/3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e/volumes"
Sep 16 00:48:13.928801 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:13.928766    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:48:13.929223 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:13.928823    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:48:15.388984 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:48:15.388940    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7m8gh" podUID="59b8ae0f-6819-4e75-8d12-27c5156bcd9d"
Sep 16 00:48:15.787000 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:15.786969    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:48:18.615880 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:18.615842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:48:18.618210 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:18.618188    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59b8ae0f-6819-4e75-8d12-27c5156bcd9d-metrics-tls\") pod \"dns-default-7m8gh\" (UID: \"59b8ae0f-6819-4e75-8d12-27c5156bcd9d\") " pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:48:18.789951 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:18.789924    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n5l29\""
Sep 16 00:48:18.798599 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:18.798581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:48:19.066974 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:19.066952    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7m8gh"]
Sep 16 00:48:19.069276 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:48:19.069251    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b8ae0f_6819_4e75_8d12_27c5156bcd9d.slice/crio-db8c3377e0fcf42612e58535d75b75508dcc175dd8afb6a47de2a0205417b2d2 WatchSource:0}: Error finding container db8c3377e0fcf42612e58535d75b75508dcc175dd8afb6a47de2a0205417b2d2: Status 404 returned error can't find the container with id db8c3377e0fcf42612e58535d75b75508dcc175dd8afb6a47de2a0205417b2d2
Sep 16 00:48:19.802838 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:19.802803    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7m8gh" event={"ID":"59b8ae0f-6819-4e75-8d12-27c5156bcd9d","Type":"ContainerStarted","Data":"db8c3377e0fcf42612e58535d75b75508dcc175dd8afb6a47de2a0205417b2d2"}
Sep 16 00:48:20.811218 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:20.811187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7m8gh" event={"ID":"59b8ae0f-6819-4e75-8d12-27c5156bcd9d","Type":"ContainerStarted","Data":"22284bf26f0a35dec6e4b876702dfc76a9c59af2a444e6c505d1a0ea7e8cb1d5"}
Sep 16 00:48:21.818689 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:21.818653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7m8gh" event={"ID":"59b8ae0f-6819-4e75-8d12-27c5156bcd9d","Type":"ContainerStarted","Data":"3dfe7ed4a799f94ebe659cb66ea22593ce1b5075e791c786c10ff2c3de4dda5c"}
Sep 16 00:48:21.819134 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:21.818784    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:48:21.835054 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:21.835012    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7m8gh" podStartSLOduration=252.196127033 podStartE2EDuration="4m13.834997333s" totalImagesPullingTime="1.6388703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:21.833145905 +0000 UTC m=+287.535968103" watchObservedRunningTime="2026-09-16 00:48:21.834997333 +0000 UTC m=+287.537819527"
Sep 16 00:48:31.824578 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:31.824547    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7m8gh"
Sep 16 00:48:33.934121 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:33.934094    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:48:33.937821 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:33.937799    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6696656695-2x5tk"
Sep 16 00:48:34.672569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:34.672546    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 00:48:38.881626 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:38.881595    2573 generic.go:356] "Generic (PLEG): container finished" podID="99772ee8-b1af-4d1c-acd4-55ea52a1a114" containerID="ae486b558d57ebc0f7e8888a2236faa04c4bc49c57bbac01b55c3dc2ea59635f" exitCode=0
Sep 16 00:48:38.881960 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:38.881667    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t" event={"ID":"99772ee8-b1af-4d1c-acd4-55ea52a1a114","Type":"ContainerDied","Data":"ae486b558d57ebc0f7e8888a2236faa04c4bc49c57bbac01b55c3dc2ea59635f"}
Sep 16 00:48:38.881960 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:38.881948    2573 scope.go:117] "RemoveContainer" containerID="ae486b558d57ebc0f7e8888a2236faa04c4bc49c57bbac01b55c3dc2ea59635f"
Sep 16 00:48:39.887422 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:39.887380    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9982t" event={"ID":"99772ee8-b1af-4d1c-acd4-55ea52a1a114","Type":"ContainerStarted","Data":"536d3e32933d0a413b02fb2c1e82edfd9e4ee39d254b722a77a178629eff8d84"}
Sep 16 00:48:55.729714 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:55.729685    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:48:55.745841 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:55.745818    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:48:55.974913 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:48:55.974886    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:13.851352 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:13.851313    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:49:13.851930 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:13.851718    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy-web" containerID="cri-o://b1c44f780c48887ebfd35a2bb68544db34990f1b5725d29bc6ce757fdb66a650" gracePeriod=600
Sep 16 00:49:13.851930 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:13.851732    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5c6dba5b859c87f84e8082851d619c7850bdccc9eac4a5b24cdfb56fe34eafca" gracePeriod=600
Sep 16 00:49:13.851930 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:13.851743    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy" containerID="cri-o://3cfaed5200a817f8a65c343352c0f9d0981112f3c663eb6c3b66490ce8ec00fe" gracePeriod=600
Sep 16 00:49:13.851930 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:13.851730    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="config-reloader" containerID="cri-o://26fd191bd5cdcb3ebc659a28f540a91839869aa0519e3495f740ab5723d14ca0" gracePeriod=600
Sep 16 00:49:13.851930 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:13.851719    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="prometheus" containerID="cri-o://d426c0d49680eb8b8ddb97729dbcbd31ef5a8bb88e818c99fdd9046d8b2953a6" gracePeriod=600
Sep 16 00:49:13.851930 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:13.851805    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="thanos-sidecar" containerID="cri-o://3193c1e3be7bf96d87726e35cf297025f3c7c25501238870d7c6cafd36221628" gracePeriod=600
Sep 16 00:49:14.030196 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030162    2573 generic.go:356] "Generic (PLEG): container finished" podID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerID="5c6dba5b859c87f84e8082851d619c7850bdccc9eac4a5b24cdfb56fe34eafca" exitCode=0
Sep 16 00:49:14.030196 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030195    2573 generic.go:356] "Generic (PLEG): container finished" podID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerID="3cfaed5200a817f8a65c343352c0f9d0981112f3c663eb6c3b66490ce8ec00fe" exitCode=0
Sep 16 00:49:14.030388 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030205    2573 generic.go:356] "Generic (PLEG): container finished" podID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerID="b1c44f780c48887ebfd35a2bb68544db34990f1b5725d29bc6ce757fdb66a650" exitCode=0
Sep 16 00:49:14.030388 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030214    2573 generic.go:356] "Generic (PLEG): container finished" podID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerID="3193c1e3be7bf96d87726e35cf297025f3c7c25501238870d7c6cafd36221628" exitCode=0
Sep 16 00:49:14.030388 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030224    2573 generic.go:356] "Generic (PLEG): container finished" podID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerID="26fd191bd5cdcb3ebc659a28f540a91839869aa0519e3495f740ab5723d14ca0" exitCode=0
Sep 16 00:49:14.030388 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030232    2573 generic.go:356] "Generic (PLEG): container finished" podID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerID="d426c0d49680eb8b8ddb97729dbcbd31ef5a8bb88e818c99fdd9046d8b2953a6" exitCode=0
Sep 16 00:49:14.030388 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030257    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerDied","Data":"5c6dba5b859c87f84e8082851d619c7850bdccc9eac4a5b24cdfb56fe34eafca"}
Sep 16 00:49:14.030388 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerDied","Data":"3cfaed5200a817f8a65c343352c0f9d0981112f3c663eb6c3b66490ce8ec00fe"}
Sep 16 00:49:14.030388 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerDied","Data":"b1c44f780c48887ebfd35a2bb68544db34990f1b5725d29bc6ce757fdb66a650"}
Sep 16 00:49:14.030726 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerDied","Data":"3193c1e3be7bf96d87726e35cf297025f3c7c25501238870d7c6cafd36221628"}
Sep 16 00:49:14.030726 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030446    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerDied","Data":"26fd191bd5cdcb3ebc659a28f540a91839869aa0519e3495f740ab5723d14ca0"}
Sep 16 00:49:14.030726 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.030484    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerDied","Data":"d426c0d49680eb8b8ddb97729dbcbd31ef5a8bb88e818c99fdd9046d8b2953a6"}
Sep 16 00:49:14.090377 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.090357    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:14.240057 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240024    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-serving-certs-ca-bundle\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240230 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240070    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-1\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240230 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240098    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-tls-assets\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240230 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240119    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-db\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240230 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240142    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240230 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240174    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-2\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240230 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240210    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-grpc-tls\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240239    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config-out\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240273    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240298    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-thanos-prometheus-http-client-file\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240326    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-metrics-client-certs\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-0\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240411    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-kube-rbac-proxy\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240442    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt4dx\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-kube-api-access-nt4dx\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240467    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-trusted-ca-bundle\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240467    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240502    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240515    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:14.240569 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240566    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-kubelet-serving-ca-bundle\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.241133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240591    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-metrics-client-ca\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.241133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240632    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-web-config\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.241133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240664    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-tls\") pod \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\" (UID: \"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446\") "
Sep 16 00:49:14.241133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240774    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:14.241133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240940    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.241133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240957    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.241133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.240971    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.241478 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.241320    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:14.241597 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.241570    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:49:14.241739 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.241719    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:14.241890 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.241867    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:14.243497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.243238    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:49:14.243497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.243233    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config" (OuterVolumeSpecName: "config") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.243497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.243468    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.243497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.243482    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.244016 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.243886    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:49:14.244675 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.244641    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.244675 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.244662    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.244852 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.244834    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-kube-api-access-nt4dx" (OuterVolumeSpecName: "kube-api-access-nt4dx") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "kube-api-access-nt4dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:49:14.245397 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.245380    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.245792 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.245771    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.246146 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.246132    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config-out" (OuterVolumeSpecName: "config-out") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:49:14.246302 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.246284    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.255824 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.255806    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-web-config" (OuterVolumeSpecName: "web-config") pod "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" (UID: "86dbddd5-e888-4fc5-bb9d-1a2d04ecd446"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:49:14.342321 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342296    2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-tls-assets\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342321 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342319    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-db\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342329    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342338    2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-grpc-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342347    2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config-out\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342354    2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-config\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342362    2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342371    2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-metrics-client-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342382    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342390    2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-kube-rbac-proxy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342397    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4dx\" (UniqueName: \"kubernetes.io/projected/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-kube-api-access-nt4dx\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342406    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342414    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342422    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342431    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-configmap-metrics-client-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342439    2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-web-config\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:14.342460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:14.342446    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:49:15.037131 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.037033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86dbddd5-e888-4fc5-bb9d-1a2d04ecd446","Type":"ContainerDied","Data":"0e9456fea5f660fdf55c9235cece525c0352545195cae4c485e055792711d13f"}
Sep 16 00:49:15.037131 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.037068    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.037629 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.037131    2573 scope.go:117] "RemoveContainer" containerID="5c6dba5b859c87f84e8082851d619c7850bdccc9eac4a5b24cdfb56fe34eafca"
Sep 16 00:49:15.047051 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.047034    2573 scope.go:117] "RemoveContainer" containerID="3cfaed5200a817f8a65c343352c0f9d0981112f3c663eb6c3b66490ce8ec00fe"
Sep 16 00:49:15.053733 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.053717    2573 scope.go:117] "RemoveContainer" containerID="b1c44f780c48887ebfd35a2bb68544db34990f1b5725d29bc6ce757fdb66a650"
Sep 16 00:49:15.057230 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.057211    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:49:15.061296 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.061278    2573 scope.go:117] "RemoveContainer" containerID="3193c1e3be7bf96d87726e35cf297025f3c7c25501238870d7c6cafd36221628"
Sep 16 00:49:15.061365 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.061315    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:49:15.067909 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.067892    2573 scope.go:117] "RemoveContainer" containerID="26fd191bd5cdcb3ebc659a28f540a91839869aa0519e3495f740ab5723d14ca0"
Sep 16 00:49:15.074451 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.074436    2573 scope.go:117] "RemoveContainer" containerID="d426c0d49680eb8b8ddb97729dbcbd31ef5a8bb88e818c99fdd9046d8b2953a6"
Sep 16 00:49:15.081232 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.081215    2573 scope.go:117] "RemoveContainer" containerID="06c9e9f0e8963ce9bb793feba5de5cf617816399c5929076b58fa5cd20dc4655"
Sep 16 00:49:15.084439 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.084408    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:49:15.085010 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.084987    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" containerName="registry"
Sep 16 00:49:15.085010 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085011    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" containerName="registry"
Sep 16 00:49:15.085135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085028    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy-web"
Sep 16 00:49:15.085135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085036    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy-web"
Sep 16 00:49:15.085135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085086    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="init-config-reloader"
Sep 16 00:49:15.085135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085096    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="init-config-reloader"
Sep 16 00:49:15.085135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085105    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy-thanos"
Sep 16 00:49:15.085135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085113    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy-thanos"
Sep 16 00:49:15.085135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085123    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="prometheus"
Sep 16 00:49:15.085135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085132    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="prometheus"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085142    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="config-reloader"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085151    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="config-reloader"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085164    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="thanos-sidecar"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085171    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="thanos-sidecar"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085181    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085188    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085257    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6b3db6-f4f1-4616-bc1c-ee9e0d43492e" containerName="registry"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085269    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="config-reloader"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085279    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="thanos-sidecar"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085286    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy-web"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085295    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085302    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="kube-rbac-proxy-thanos"
Sep 16 00:49:15.085393 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.085310    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" containerName="prometheus"
Sep 16 00:49:15.090435 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.090421    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.092683 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.092664    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 00:49:15.092804 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.092788    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 00:49:15.092979 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.092961    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 00:49:15.092979 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.092971    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fcqh6uqer9qrk\""
Sep 16 00:49:15.093204 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093189    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 00:49:15.093280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093191    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 00:49:15.093355 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093340    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h7t6g\""
Sep 16 00:49:15.093412 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093338    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 00:49:15.093543 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093495    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 00:49:15.093543 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093498    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 00:49:15.093707 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093552    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 00:49:15.093707 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093498    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 00:49:15.093707 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093696    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 00:49:15.093895 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.093872    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 00:49:15.099386 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.099364    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 00:49:15.101677 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.101654    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 00:49:15.104341 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.104320    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:49:15.147869 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.147842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.147869 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.147870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148009 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.147900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148009 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.147917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148009 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.147933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148009 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.147960    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148009 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.147985    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bb6v\" (UniqueName: \"kubernetes.io/projected/5c201f19-8f74-42e7-9367-18d2fc0045fb-kube-api-access-6bb6v\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148152 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-config\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148152 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148152 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148097    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148152 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148113    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148152 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148294 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148158    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148294 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148294 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148294 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148251    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148294 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148437 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148298    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c201f19-8f74-42e7-9367-18d2fc0045fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148437 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c201f19-8f74-42e7-9367-18d2fc0045fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.148437 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.148333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249129 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249129 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249135    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249298 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249151    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249298 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249172    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c201f19-8f74-42e7-9367-18d2fc0045fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249298 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249288    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c201f19-8f74-42e7-9367-18d2fc0045fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249439 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249439 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249378    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249439 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249404    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249626 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249626 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249626 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249626 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249626 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bb6v\" (UniqueName: \"kubernetes.io/projected/5c201f19-8f74-42e7-9367-18d2fc0045fb-kube-api-access-6bb6v\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249626 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-config\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249929 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249929 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249929 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249929 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249929 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.249929 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.249786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.250392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.250174    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.250392 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.250353    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.250498 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.250478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.251106 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.251080    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.252540 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.252505    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c201f19-8f74-42e7-9367-18d2fc0045fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.252746 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.252719    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.253150 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.252992    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c201f19-8f74-42e7-9367-18d2fc0045fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.253247 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.253230    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.253316 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.253248    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.254135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.253488    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.254135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.253497    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.254135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.253932    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.254135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.254023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.254135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.254097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c201f19-8f74-42e7-9367-18d2fc0045fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.254558 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.254197    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-config\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.255129 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.255103    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.255474 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.255449    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.256021 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.255998    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.256257 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.256241    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c201f19-8f74-42e7-9367-18d2fc0045fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.257772 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.257753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bb6v\" (UniqueName: \"kubernetes.io/projected/5c201f19-8f74-42e7-9367-18d2fc0045fb-kube-api-access-6bb6v\") pod \"prometheus-k8s-0\" (UID: \"5c201f19-8f74-42e7-9367-18d2fc0045fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.403996 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.403917    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:49:15.688110 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:15.688087    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 00:49:15.690625 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:49:15.690600    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c201f19_8f74_42e7_9367_18d2fc0045fb.slice/crio-061fe22d86c1198e7dc9d8c73b83a7ef7f20b4296d834225c6e83d7a8fe719d3 WatchSource:0}: Error finding container 061fe22d86c1198e7dc9d8c73b83a7ef7f20b4296d834225c6e83d7a8fe719d3: Status 404 returned error can't find the container with id 061fe22d86c1198e7dc9d8c73b83a7ef7f20b4296d834225c6e83d7a8fe719d3
Sep 16 00:49:16.042882 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:16.042841    2573 generic.go:356] "Generic (PLEG): container finished" podID="5c201f19-8f74-42e7-9367-18d2fc0045fb" containerID="3fad24b192794b728791a0e08b9a30366e2c7fb016991a85f66d4106637e2242" exitCode=0
Sep 16 00:49:16.043271 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:16.042902    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c201f19-8f74-42e7-9367-18d2fc0045fb","Type":"ContainerDied","Data":"3fad24b192794b728791a0e08b9a30366e2c7fb016991a85f66d4106637e2242"}
Sep 16 00:49:16.043271 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:16.042975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c201f19-8f74-42e7-9367-18d2fc0045fb","Type":"ContainerStarted","Data":"061fe22d86c1198e7dc9d8c73b83a7ef7f20b4296d834225c6e83d7a8fe719d3"}
Sep 16 00:49:16.769982 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:16.769950    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86dbddd5-e888-4fc5-bb9d-1a2d04ecd446" path="/var/lib/kubelet/pods/86dbddd5-e888-4fc5-bb9d-1a2d04ecd446/volumes"
Sep 16 00:49:17.055486 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:17.055400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c201f19-8f74-42e7-9367-18d2fc0045fb","Type":"ContainerStarted","Data":"ad96e9ceba9b25b4d4dcd0ea665bc3a6e86f00b301885e676dc1a807f0b0609e"}
Sep 16 00:49:17.055872 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:17.055552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c201f19-8f74-42e7-9367-18d2fc0045fb","Type":"ContainerStarted","Data":"6011eaa78ce68c43a54b35f16e907b14b818f914f858ecf4478217c92645abee"}
Sep 16 00:49:17.055872 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:17.055596    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c201f19-8f74-42e7-9367-18d2fc0045fb","Type":"ContainerStarted","Data":"47adea7ab8b445f73dfbc086af570c7c2bf3b9ff0d731f57ee676cb55a0c7024"}
Sep 16 00:49:17.055872 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:17.055654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c201f19-8f74-42e7-9367-18d2fc0045fb","Type":"ContainerStarted","Data":"92a2fac82d8c6180ee6f3b6da829c2d64ded5fd0f2f5cc3b9f46b5e90e7434ee"}
Sep 16 00:49:17.055872 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:17.055704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c201f19-8f74-42e7-9367-18d2fc0045fb","Type":"ContainerStarted","Data":"ab153c1724f0038a1835d1118fb21c471cbe28f6698eaab9077b2ba9d74a5a41"}
Sep 16 00:49:17.055872 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:17.055743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c201f19-8f74-42e7-9367-18d2fc0045fb","Type":"ContainerStarted","Data":"08e3c8365f44596e7ebdc37c74f157ba4bb1da9ffd5e4420a5952dbacc4436bc"}
Sep 16 00:49:17.080939 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:17.080882    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.059362491 podStartE2EDuration="2.080865459s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.502968ms" isStatefulPod=false podCreationTimestamp="2026-09-16 00:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:49:17.079103964 +0000 UTC m=+342.781926181" watchObservedRunningTime="2026-09-16 00:49:17.080865459 +0000 UTC m=+342.783687653"
Sep 16 00:49:20.404261 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:49:20.404219    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:50:15.404372 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:50:15.404344    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:50:15.420321 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:50:15.420299    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:50:16.294327 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:50:16.294300    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 00:52:19.368005 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.367973    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"]
Sep 16 00:52:19.371382 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.371362    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.374773 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.374733    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:52:19.374902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.374733    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 00:52:19.374902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.374735    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 00:52:19.374902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.374736    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 00:52:19.374902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.374862    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 00:52:19.375215 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.375179    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-79rlj\""
Sep 16 00:52:19.390645 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.390617    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"]
Sep 16 00:52:19.464778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.464748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dbca285-f758-413f-929b-280070f09d27-cert\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.464928 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.464782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dbca285-f758-413f-929b-280070f09d27-metrics-cert\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.464928 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.464806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgpx\" (UniqueName: \"kubernetes.io/projected/2dbca285-f758-413f-929b-280070f09d27-kube-api-access-9zgpx\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.464928 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.464871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2dbca285-f758-413f-929b-280070f09d27-manager-config\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.565974 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.565945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgpx\" (UniqueName: \"kubernetes.io/projected/2dbca285-f758-413f-929b-280070f09d27-kube-api-access-9zgpx\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.566114 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.566005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2dbca285-f758-413f-929b-280070f09d27-manager-config\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.566114 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.566030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dbca285-f758-413f-929b-280070f09d27-cert\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.566114 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.566051    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dbca285-f758-413f-929b-280070f09d27-metrics-cert\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.566772 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.566750    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2dbca285-f758-413f-929b-280070f09d27-manager-config\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.568596 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.568570    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dbca285-f758-413f-929b-280070f09d27-cert\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.568677 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.568603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dbca285-f758-413f-929b-280070f09d27-metrics-cert\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.578588 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.578568    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgpx\" (UniqueName: \"kubernetes.io/projected/2dbca285-f758-413f-929b-280070f09d27-kube-api-access-9zgpx\") pod \"lws-controller-manager-54b79494c6-4h7nt\" (UID: \"2dbca285-f758-413f-929b-280070f09d27\") " pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.680228 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.680147    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:19.957347 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.957323    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"]
Sep 16 00:52:19.959621 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:52:19.959592    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbca285_f758_413f_929b_280070f09d27.slice/crio-26837446a782bbf1f9b45af735aec47c2a2b985eaf4f8186dcd740d8740db26f WatchSource:0}: Error finding container 26837446a782bbf1f9b45af735aec47c2a2b985eaf4f8186dcd740d8740db26f: Status 404 returned error can't find the container with id 26837446a782bbf1f9b45af735aec47c2a2b985eaf4f8186dcd740d8740db26f
Sep 16 00:52:19.961687 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:19.961670    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:52:20.748492 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:20.748444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt" event={"ID":"2dbca285-f758-413f-929b-280070f09d27","Type":"ContainerStarted","Data":"26837446a782bbf1f9b45af735aec47c2a2b985eaf4f8186dcd740d8740db26f"}
Sep 16 00:52:22.434152 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.431622    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"]
Sep 16 00:52:22.436676 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.436653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.438403 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.438381    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 00:52:22.438877 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.438850    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cqdqc\""
Sep 16 00:52:22.439106 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.439066    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 00:52:22.439216 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.439108    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 00:52:22.439283 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.439240    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 00:52:22.449701 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.449673    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"]
Sep 16 00:52:22.488021 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.487987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.488179 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.488055    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-tmp\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.488179 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.488099    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-manifests\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.488307 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.488234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fdv\" (UniqueName: \"kubernetes.io/projected/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-kube-api-access-q9fdv\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.488307 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.488272    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.589066 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.589045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fdv\" (UniqueName: \"kubernetes.io/projected/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-kube-api-access-q9fdv\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.589146 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.589079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.589146 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.589119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.589226 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.589157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-tmp\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.589226 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.589183    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-manifests\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.589608 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.589587    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-manifests\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.592008 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.591981    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-tmp\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.592096 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.592032    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.592175 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.592139    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.596723 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.596704    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fdv\" (UniqueName: \"kubernetes.io/projected/f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c-kube-api-access-q9fdv\") pod \"opendatahub-operator-controller-manager-9778fc5bd-hnmlh\" (UID: \"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c\") " pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.750201 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.750173    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:22.759208 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.759184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt" event={"ID":"2dbca285-f758-413f-929b-280070f09d27","Type":"ContainerStarted","Data":"48d3668e105b9179638b4c9e83911a33bcfa1d088328599b1d0ddeed02af0c44"}
Sep 16 00:52:22.759324 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.759301    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:22.780276 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:22.780226    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt" podStartSLOduration=1.170698579 podStartE2EDuration="3.780209078s" totalImagesPullingTime="2.609510499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:22.778467336 +0000 UTC m=+528.481289545" watchObservedRunningTime="2026-09-16 00:52:22.780209078 +0000 UTC m=+528.483031273"
Sep 16 00:52:23.036322 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:23.036297    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"]
Sep 16 00:52:23.039433 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:52:23.039403    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fc9abd_63d9_47a4_a8b6_bdfbf76b6c6c.slice/crio-44e8a1bbe468613db632d2602467cd3716e12ecefc9ee70fa5ec120f67d892c8 WatchSource:0}: Error finding container 44e8a1bbe468613db632d2602467cd3716e12ecefc9ee70fa5ec120f67d892c8: Status 404 returned error can't find the container with id 44e8a1bbe468613db632d2602467cd3716e12ecefc9ee70fa5ec120f67d892c8
Sep 16 00:52:23.767112 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:23.767075    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh" event={"ID":"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c","Type":"ContainerStarted","Data":"44e8a1bbe468613db632d2602467cd3716e12ecefc9ee70fa5ec120f67d892c8"}
Sep 16 00:52:26.781967 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:26.781937    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh" event={"ID":"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c","Type":"ContainerStarted","Data":"c7b596cdcd028ead937d0fccabb9fec3c17247a2a6ce611f0b412ebea3396272"}
Sep 16 00:52:27.788108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:27.788070    2573 generic.go:356] "Generic (PLEG): container finished" podID="f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c" containerID="c7b596cdcd028ead937d0fccabb9fec3c17247a2a6ce611f0b412ebea3396272" exitCode=0
Sep 16 00:52:27.788468 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:27.788163    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh" event={"ID":"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c","Type":"ContainerDied","Data":"c7b596cdcd028ead937d0fccabb9fec3c17247a2a6ce611f0b412ebea3396272"}
Sep 16 00:52:28.793911 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:28.793880    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh" event={"ID":"f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c","Type":"ContainerStarted","Data":"a7acdf1ff1f583cae6d08b17ce8fb0b22dff9ddc909b8210adc1f19aa26ba9eb"}
Sep 16 00:52:28.794267 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:28.794007    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:28.814458 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:28.814415    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh" podStartSLOduration=2.476032454 podStartE2EDuration="6.814400055s" totalImagesPullingTime="3.296543904s" totalInitContainerRuntime="1.041823697s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:28.812100245 +0000 UTC m=+534.514922430" watchObservedRunningTime="2026-09-16 00:52:28.814400055 +0000 UTC m=+534.517222249"
Sep 16 00:52:33.769754 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:33.769722    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54b79494c6-4h7nt"
Sep 16 00:52:39.800781 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:39.800747    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9778fc5bd-hnmlh"
Sep 16 00:52:49.107085 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.107052    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"]
Sep 16 00:52:49.121215 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.121184    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.122409 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.122375    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"]
Sep 16 00:52:49.123360 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.123333    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-f9mj9\""
Sep 16 00:52:49.123500 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.123384    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 00:52:49.195456 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195429    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cn5\" (UniqueName: \"kubernetes.io/projected/4fa70455-2958-412f-85db-05e6c74c80d0-kube-api-access-p9cn5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.195584 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.195584 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195482    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa70455-2958-412f-85db-05e6c74c80d0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.195656 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.195656 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa70455-2958-412f-85db-05e6c74c80d0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.195656 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.195796 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195693    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.195796 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195709    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa70455-2958-412f-85db-05e6c74c80d0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.195796 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.195730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296753 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296715    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296753 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa70455-2958-412f-85db-05e6c74c80d0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296933 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296933 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa70455-2958-412f-85db-05e6c74c80d0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296933 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296833    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296933 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296933 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa70455-2958-412f-85db-05e6c74c80d0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296933 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.296933 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.296933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cn5\" (UniqueName: \"kubernetes.io/projected/4fa70455-2958-412f-85db-05e6c74c80d0-kube-api-access-p9cn5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.297291 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.297137    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.297351 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.297298    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.297351 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.297309    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.297461 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.297392    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.297639 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.297618    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa70455-2958-412f-85db-05e6c74c80d0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.299169 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.299149    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa70455-2958-412f-85db-05e6c74c80d0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.299572 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.299555    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa70455-2958-412f-85db-05e6c74c80d0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.306668 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.306640    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa70455-2958-412f-85db-05e6c74c80d0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.306776 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.306665    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cn5\" (UniqueName: \"kubernetes.io/projected/4fa70455-2958-412f-85db-05e6c74c80d0-kube-api-access-p9cn5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c92k86\" (UID: \"4fa70455-2958-412f-85db-05e6c74c80d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.433564 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.433467    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:49.709838 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.709812    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"]
Sep 16 00:52:49.711902 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:52:49.711875    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa70455_2958_412f_85db_05e6c74c80d0.slice/crio-94ed196e81751a0fcfa8c677c7800af591dbd8c3353694f8a00464a88f36fbe4 WatchSource:0}: Error finding container 94ed196e81751a0fcfa8c677c7800af591dbd8c3353694f8a00464a88f36fbe4: Status 404 returned error can't find the container with id 94ed196e81751a0fcfa8c677c7800af591dbd8c3353694f8a00464a88f36fbe4
Sep 16 00:52:49.881365 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:49.881330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86" event={"ID":"4fa70455-2958-412f-85db-05e6c74c80d0","Type":"ContainerStarted","Data":"94ed196e81751a0fcfa8c677c7800af591dbd8c3353694f8a00464a88f36fbe4"}
Sep 16 00:52:52.668854 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.668817    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kgj5q"]
Sep 16 00:52:52.675729 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.675705    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:52.678355 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.678325    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 00:52:52.678491 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.678363    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-q78tw\""
Sep 16 00:52:52.685834 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.685810    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kgj5q"]
Sep 16 00:52:52.726876 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.726846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pps8n\" (UniqueName: \"kubernetes.io/projected/18c757f2-5e60-4944-bd69-b1ae2ae3b647-kube-api-access-pps8n\") pod \"odh-model-controller-7998c67dc4-kgj5q\" (UID: \"18c757f2-5e60-4944-bd69-b1ae2ae3b647\") " pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:52.727029 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.726924    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18c757f2-5e60-4944-bd69-b1ae2ae3b647-cert\") pod \"odh-model-controller-7998c67dc4-kgj5q\" (UID: \"18c757f2-5e60-4944-bd69-b1ae2ae3b647\") " pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:52.828143 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.828108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pps8n\" (UniqueName: \"kubernetes.io/projected/18c757f2-5e60-4944-bd69-b1ae2ae3b647-kube-api-access-pps8n\") pod \"odh-model-controller-7998c67dc4-kgj5q\" (UID: \"18c757f2-5e60-4944-bd69-b1ae2ae3b647\") " pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:52.828323 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.828209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18c757f2-5e60-4944-bd69-b1ae2ae3b647-cert\") pod \"odh-model-controller-7998c67dc4-kgj5q\" (UID: \"18c757f2-5e60-4944-bd69-b1ae2ae3b647\") " pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:52.831474 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.831448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18c757f2-5e60-4944-bd69-b1ae2ae3b647-cert\") pod \"odh-model-controller-7998c67dc4-kgj5q\" (UID: \"18c757f2-5e60-4944-bd69-b1ae2ae3b647\") " pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:52.840239 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.840217    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pps8n\" (UniqueName: \"kubernetes.io/projected/18c757f2-5e60-4944-bd69-b1ae2ae3b647-kube-api-access-pps8n\") pod \"odh-model-controller-7998c67dc4-kgj5q\" (UID: \"18c757f2-5e60-4944-bd69-b1ae2ae3b647\") " pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:52.988606 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:52.988575    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:53.080628 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.080587    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:53.080884 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.080669    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:53.080884 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.080706    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:53.278436 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.278347    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kgj5q"]
Sep 16 00:52:53.282352 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:52:53.282321    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c757f2_5e60_4944_bd69_b1ae2ae3b647.slice/crio-54d221b45b275e101dc79c8be1cfc12d38f9f3d8aee812d215546771e21ae56d WatchSource:0}: Error finding container 54d221b45b275e101dc79c8be1cfc12d38f9f3d8aee812d215546771e21ae56d: Status 404 returned error can't find the container with id 54d221b45b275e101dc79c8be1cfc12d38f9f3d8aee812d215546771e21ae56d
Sep 16 00:52:53.328446 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.328406    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-t7zkb"]
Sep 16 00:52:53.331976 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.331952    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:53.334300 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.334277    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 00:52:53.334420 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.334339    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vmcf5\""
Sep 16 00:52:53.343329 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.343310    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-t7zkb"]
Sep 16 00:52:53.433281 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.433255    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695cf87c-0087-451d-afba-cbcf5b67bff4-tls-certs\") pod \"model-serving-api-54d89bd5fc-t7zkb\" (UID: \"695cf87c-0087-451d-afba-cbcf5b67bff4\") " pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:53.433442 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.433302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5gc\" (UniqueName: \"kubernetes.io/projected/695cf87c-0087-451d-afba-cbcf5b67bff4-kube-api-access-jm5gc\") pod \"model-serving-api-54d89bd5fc-t7zkb\" (UID: \"695cf87c-0087-451d-afba-cbcf5b67bff4\") " pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:53.534200 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.534132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695cf87c-0087-451d-afba-cbcf5b67bff4-tls-certs\") pod \"model-serving-api-54d89bd5fc-t7zkb\" (UID: \"695cf87c-0087-451d-afba-cbcf5b67bff4\") " pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:53.534200 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.534176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5gc\" (UniqueName: \"kubernetes.io/projected/695cf87c-0087-451d-afba-cbcf5b67bff4-kube-api-access-jm5gc\") pod \"model-serving-api-54d89bd5fc-t7zkb\" (UID: \"695cf87c-0087-451d-afba-cbcf5b67bff4\") " pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:53.536675 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.536650    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695cf87c-0087-451d-afba-cbcf5b67bff4-tls-certs\") pod \"model-serving-api-54d89bd5fc-t7zkb\" (UID: \"695cf87c-0087-451d-afba-cbcf5b67bff4\") " pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:53.544468 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.544442    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5gc\" (UniqueName: \"kubernetes.io/projected/695cf87c-0087-451d-afba-cbcf5b67bff4-kube-api-access-jm5gc\") pod \"model-serving-api-54d89bd5fc-t7zkb\" (UID: \"695cf87c-0087-451d-afba-cbcf5b67bff4\") " pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:53.642069 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.642046    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:53.904175 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.904093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86" event={"ID":"4fa70455-2958-412f-85db-05e6c74c80d0","Type":"ContainerStarted","Data":"23aa76ca8b40c8bd9f351e7ffcbcfa7d568217adc9a6d921b6c710cae2892b89"}
Sep 16 00:52:53.905399 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.905374    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q" event={"ID":"18c757f2-5e60-4944-bd69-b1ae2ae3b647","Type":"ContainerStarted","Data":"54d221b45b275e101dc79c8be1cfc12d38f9f3d8aee812d215546771e21ae56d"}
Sep 16 00:52:53.911449 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.911426    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-t7zkb"]
Sep 16 00:52:53.913046 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:52:53.913026    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695cf87c_0087_451d_afba_cbcf5b67bff4.slice/crio-92975ecaadbe9591a00f00861e46736fd63a926dfe6bdb49b994123c2b7d2401 WatchSource:0}: Error finding container 92975ecaadbe9591a00f00861e46736fd63a926dfe6bdb49b994123c2b7d2401: Status 404 returned error can't find the container with id 92975ecaadbe9591a00f00861e46736fd63a926dfe6bdb49b994123c2b7d2401
Sep 16 00:52:53.925555 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:53.925496    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86" podStartSLOduration=1.559102619 podStartE2EDuration="4.925481115s" totalImagesPullingTime="3.366378496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:53.923444818 +0000 UTC m=+559.626267013" watchObservedRunningTime="2026-09-16 00:52:53.925481115 +0000 UTC m=+559.628303309"
Sep 16 00:52:54.434608 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:54.434573    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:54.439446 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:54.439422    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:54.914182 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:54.914141    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb" event={"ID":"695cf87c-0087-451d-afba-cbcf5b67bff4","Type":"ContainerStarted","Data":"92975ecaadbe9591a00f00861e46736fd63a926dfe6bdb49b994123c2b7d2401"}
Sep 16 00:52:54.914672 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:54.914407    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:54.915565 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:54.915503    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c92k86"
Sep 16 00:52:57.936717 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:57.936624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q" event={"ID":"18c757f2-5e60-4944-bd69-b1ae2ae3b647","Type":"ContainerStarted","Data":"697711c05a8ddd4bc86b785bfcdaaaf04399b0460f32ac753f5aa36120a8e421"}
Sep 16 00:52:57.937205 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:57.936759    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:52:57.938838 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:57.938811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb" event={"ID":"695cf87c-0087-451d-afba-cbcf5b67bff4","Type":"ContainerStarted","Data":"01935815149ecd4fb1091caccbb0ff4e69ad66e0548d4dd7eab3321ed2ecdf5c"}
Sep 16 00:52:57.938982 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:57.938965    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:52:57.955339 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:57.955297    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q" podStartSLOduration=1.67024279 podStartE2EDuration="5.95528316s" totalImagesPullingTime="4.28504037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:57.954457003 +0000 UTC m=+563.657279199" watchObservedRunningTime="2026-09-16 00:52:57.95528316 +0000 UTC m=+563.658105354"
Sep 16 00:52:57.969977 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:52:57.969933    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb" podStartSLOduration=1.278938149 podStartE2EDuration="4.96992012s" totalImagesPullingTime="3.690981971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:57.969048364 +0000 UTC m=+563.671870559" watchObservedRunningTime="2026-09-16 00:52:57.96992012 +0000 UTC m=+563.672742315"
Sep 16 00:53:02.101103 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.101063    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"]
Sep 16 00:53:02.104487 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.104471    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:02.107380 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.107359    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wr4zc\""
Sep 16 00:53:02.107846 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.107829    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 00:53:02.117213 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.117180    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"]
Sep 16 00:53:02.120479 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.120458    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"]
Sep 16 00:53:02.120602 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.120564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:02.122756 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.122735    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 00:53:02.122849 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.122753    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sngvb\""
Sep 16 00:53:02.133267 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.133241    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"]
Sep 16 00:53:02.214272 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.214232    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1-cert\") pod \"kserve-controller-manager-d6f4b65b8-9zc66\" (UID: \"c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:02.214272 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.214276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkp4\" (UniqueName: \"kubernetes.io/projected/c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1-kube-api-access-7nkp4\") pod \"kserve-controller-manager-d6f4b65b8-9zc66\" (UID: \"c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:02.214479 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.214388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4sq\" (UniqueName: \"kubernetes.io/projected/1497812b-fa7c-4aa9-9270-7d60bb287164-kube-api-access-lm4sq\") pod \"llmisvc-controller-manager-777c6bd7d5-6gvb5\" (UID: \"1497812b-fa7c-4aa9-9270-7d60bb287164\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:02.214479 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.214433    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1497812b-fa7c-4aa9-9270-7d60bb287164-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6gvb5\" (UID: \"1497812b-fa7c-4aa9-9270-7d60bb287164\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:02.315656 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.315618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1497812b-fa7c-4aa9-9270-7d60bb287164-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6gvb5\" (UID: \"1497812b-fa7c-4aa9-9270-7d60bb287164\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:02.315838 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.315674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1-cert\") pod \"kserve-controller-manager-d6f4b65b8-9zc66\" (UID: \"c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:02.315838 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.315695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkp4\" (UniqueName: \"kubernetes.io/projected/c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1-kube-api-access-7nkp4\") pod \"kserve-controller-manager-d6f4b65b8-9zc66\" (UID: \"c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:02.315838 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.315750    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4sq\" (UniqueName: \"kubernetes.io/projected/1497812b-fa7c-4aa9-9270-7d60bb287164-kube-api-access-lm4sq\") pod \"llmisvc-controller-manager-777c6bd7d5-6gvb5\" (UID: \"1497812b-fa7c-4aa9-9270-7d60bb287164\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:02.315838 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:53:02.315761    2573 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 00:53:02.315838 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:53:02.315837    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1497812b-fa7c-4aa9-9270-7d60bb287164-cert podName:1497812b-fa7c-4aa9-9270-7d60bb287164 nodeName:}" failed. No retries permitted until 2026-09-16 00:53:02.815819569 +0000 UTC m=+568.518641746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1497812b-fa7c-4aa9-9270-7d60bb287164-cert") pod "llmisvc-controller-manager-777c6bd7d5-6gvb5" (UID: "1497812b-fa7c-4aa9-9270-7d60bb287164") : secret "llmisvc-webhook-server-cert" not found
Sep 16 00:53:02.318263 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.318243    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1-cert\") pod \"kserve-controller-manager-d6f4b65b8-9zc66\" (UID: \"c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:02.328484 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.328461    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkp4\" (UniqueName: \"kubernetes.io/projected/c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1-kube-api-access-7nkp4\") pod \"kserve-controller-manager-d6f4b65b8-9zc66\" (UID: \"c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:02.328611 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.328516    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4sq\" (UniqueName: \"kubernetes.io/projected/1497812b-fa7c-4aa9-9270-7d60bb287164-kube-api-access-lm4sq\") pod \"llmisvc-controller-manager-777c6bd7d5-6gvb5\" (UID: \"1497812b-fa7c-4aa9-9270-7d60bb287164\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:02.415045 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.414959    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:02.687821 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.687755    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"]
Sep 16 00:53:02.690508 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:53:02.690484    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a716f3_27b4_40ce_9a6a_d1ef0b3bd2c1.slice/crio-5c2d727bffa0f37f5fa1c024495c5215d9b0e049985e9784595743dff3f8ed4e WatchSource:0}: Error finding container 5c2d727bffa0f37f5fa1c024495c5215d9b0e049985e9784595743dff3f8ed4e: Status 404 returned error can't find the container with id 5c2d727bffa0f37f5fa1c024495c5215d9b0e049985e9784595743dff3f8ed4e
Sep 16 00:53:02.818659 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.818631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1497812b-fa7c-4aa9-9270-7d60bb287164-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6gvb5\" (UID: \"1497812b-fa7c-4aa9-9270-7d60bb287164\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:02.821097 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.821080    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1497812b-fa7c-4aa9-9270-7d60bb287164-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6gvb5\" (UID: \"1497812b-fa7c-4aa9-9270-7d60bb287164\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:02.960916 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:02.960885    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66" event={"ID":"c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1","Type":"ContainerStarted","Data":"5c2d727bffa0f37f5fa1c024495c5215d9b0e049985e9784595743dff3f8ed4e"}
Sep 16 00:53:03.030142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:03.030113    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:03.303760 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:03.303736    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"]
Sep 16 00:53:03.305998 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:53:03.305972    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1497812b_fa7c_4aa9_9270_7d60bb287164.slice/crio-889098ac73980769973ae80de6a56b0e3f8cd448c3d4d966eb9058ca6c169e4f WatchSource:0}: Error finding container 889098ac73980769973ae80de6a56b0e3f8cd448c3d4d966eb9058ca6c169e4f: Status 404 returned error can't find the container with id 889098ac73980769973ae80de6a56b0e3f8cd448c3d4d966eb9058ca6c169e4f
Sep 16 00:53:03.966974 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:03.966942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5" event={"ID":"1497812b-fa7c-4aa9-9270-7d60bb287164","Type":"ContainerStarted","Data":"889098ac73980769973ae80de6a56b0e3f8cd448c3d4d966eb9058ca6c169e4f"}
Sep 16 00:53:06.983418 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:06.983377    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66" event={"ID":"c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1","Type":"ContainerStarted","Data":"0c384d08738c2760bd14f5189fb608447511e5b06fef7a75021ef61f69e93927"}
Sep 16 00:53:06.983890 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:06.983502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:53:06.985076 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:06.985054    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5" event={"ID":"1497812b-fa7c-4aa9-9270-7d60bb287164","Type":"ContainerStarted","Data":"1185758e5d98872beb18acb41c649346e0cd954335eba8c3f8fa049be6e15c95"}
Sep 16 00:53:06.985151 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:06.985105    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:06.998898 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:06.998860    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66" podStartSLOduration=2.152060303 podStartE2EDuration="4.998847624s" totalImagesPullingTime="2.846787321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:06.998139127 +0000 UTC m=+572.700961323" watchObservedRunningTime="2026-09-16 00:53:06.998847624 +0000 UTC m=+572.701669819"
Sep 16 00:53:07.014484 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:07.014445    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5" podStartSLOduration=2.028311462 podStartE2EDuration="5.014434478s" totalImagesPullingTime="2.986123016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:07.013317835 +0000 UTC m=+572.716140031" watchObservedRunningTime="2026-09-16 00:53:07.014434478 +0000 UTC m=+572.717256673"
Sep 16 00:53:08.945039 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:08.945012    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kgj5q"
Sep 16 00:53:08.947060 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:08.947042    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-t7zkb"
Sep 16 00:53:37.992686 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:37.992656    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6gvb5"
Sep 16 00:53:37.995874 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:53:37.995848    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9zc66"
Sep 16 00:54:18.646142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.646109    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"]
Sep 16 00:54:18.649699 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.649680    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:18.651935 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.651915    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wsn7n\""
Sep 16 00:54:18.651935 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.651928    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 00:54:18.652111 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.651957    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 00:54:18.663469 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.663447    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"]
Sep 16 00:54:18.828653 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.828616    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" (UID: \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:18.828833 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.828673    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw64r\" (UniqueName: \"kubernetes.io/projected/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-kube-api-access-cw64r\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" (UID: \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:18.929716 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.929636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" (UID: \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:18.929716 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.929705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw64r\" (UniqueName: \"kubernetes.io/projected/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-kube-api-access-cw64r\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" (UID: \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:18.929997 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.929975    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" (UID: \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:18.938655 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.938628    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw64r\" (UniqueName: \"kubernetes.io/projected/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-kube-api-access-cw64r\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" (UID: \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:18.960636 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:18.960612    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:19.241775 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:19.241656    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"]
Sep 16 00:54:19.244257 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:54:19.244233    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5a82e1_1390_4e10_aa6d_d84ae77dcd16.slice/crio-6531090fed799296470f9f08932c8a775866d2513f75d5e293c86e1a6ae86b2d WatchSource:0}: Error finding container 6531090fed799296470f9f08932c8a775866d2513f75d5e293c86e1a6ae86b2d: Status 404 returned error can't find the container with id 6531090fed799296470f9f08932c8a775866d2513f75d5e293c86e1a6ae86b2d
Sep 16 00:54:19.324809 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:19.324784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t" event={"ID":"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16","Type":"ContainerStarted","Data":"6531090fed799296470f9f08932c8a775866d2513f75d5e293c86e1a6ae86b2d"}
Sep 16 00:54:24.353436 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:24.353396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t" event={"ID":"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16","Type":"ContainerStarted","Data":"3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0"}
Sep 16 00:54:24.353824 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:24.353467    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:24.374139 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:24.374089    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t" podStartSLOduration=1.680599034 podStartE2EDuration="6.374072671s" totalImagesPullingTime="4.693473637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:54:24.371769574 +0000 UTC m=+650.074591770" watchObservedRunningTime="2026-09-16 00:54:24.374072671 +0000 UTC m=+650.076894866"
Sep 16 00:54:35.360875 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:35.360847    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:37.570000 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.569962    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"]
Sep 16 00:54:37.570482 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.570237    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" containerName="manager" containerID="cri-o://3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0" gracePeriod=2
Sep 16 00:54:37.581188 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.581162    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"]
Sep 16 00:54:37.585117 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.585097    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"]
Sep 16 00:54:37.585231 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.585202    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:37.594206 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:37.594160    2573 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\": pods \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:37.595874 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:37.595842    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" is forbidden: User \"system:node:ip-10-0-131-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-89.ec2.internal' and this object" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:37.597001 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.596981    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"]
Sep 16 00:54:37.597304 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.597291    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" containerName="manager"
Sep 16 00:54:37.597354 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.597305    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" containerName="manager"
Sep 16 00:54:37.597388 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.597371    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" containerName="manager"
Sep 16 00:54:37.600892 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.600864    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:37.603225 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.603203    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"]
Sep 16 00:54:37.615938 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.615920    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"]
Sep 16 00:54:37.628118 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:37.628090    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" is forbidden: User \"system:node:ip-10-0-131-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-89.ec2.internal' and this object" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:37.691417 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.691391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/057d261b-74f4-4e5f-97ca-b89abff9c4e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6pddm\" (UID: \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:37.691552 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.691428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhft\" (UniqueName: \"kubernetes.io/projected/057d261b-74f4-4e5f-97ca-b89abff9c4e5-kube-api-access-xjhft\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6pddm\" (UID: \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:37.691552 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.691448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9468dff9-8f91-4873-9127-90317ab35382-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-95ztx\" (UID: \"9468dff9-8f91-4873-9127-90317ab35382\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:37.691552 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.691467    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrm6\" (UniqueName: \"kubernetes.io/projected/9468dff9-8f91-4873-9127-90317ab35382-kube-api-access-rsrm6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-95ztx\" (UID: \"9468dff9-8f91-4873-9127-90317ab35382\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:37.792095 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.792056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/057d261b-74f4-4e5f-97ca-b89abff9c4e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6pddm\" (UID: \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:37.792236 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.792110    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhft\" (UniqueName: \"kubernetes.io/projected/057d261b-74f4-4e5f-97ca-b89abff9c4e5-kube-api-access-xjhft\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6pddm\" (UID: \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:37.792300 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.792232    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9468dff9-8f91-4873-9127-90317ab35382-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-95ztx\" (UID: \"9468dff9-8f91-4873-9127-90317ab35382\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:37.792300 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.792281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrm6\" (UniqueName: \"kubernetes.io/projected/9468dff9-8f91-4873-9127-90317ab35382-kube-api-access-rsrm6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-95ztx\" (UID: \"9468dff9-8f91-4873-9127-90317ab35382\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:37.792538 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.792495    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/057d261b-74f4-4e5f-97ca-b89abff9c4e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6pddm\" (UID: \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:37.792647 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.792625    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9468dff9-8f91-4873-9127-90317ab35382-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-95ztx\" (UID: \"9468dff9-8f91-4873-9127-90317ab35382\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:37.800225 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.800205    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrm6\" (UniqueName: \"kubernetes.io/projected/9468dff9-8f91-4873-9127-90317ab35382-kube-api-access-rsrm6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-95ztx\" (UID: \"9468dff9-8f91-4873-9127-90317ab35382\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:37.800332 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.800297    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhft\" (UniqueName: \"kubernetes.io/projected/057d261b-74f4-4e5f-97ca-b89abff9c4e5-kube-api-access-xjhft\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6pddm\" (UID: \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:37.819753 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.819738    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:37.821502 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:37.821461    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" is forbidden: User \"system:node:ip-10-0-131-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-89.ec2.internal' and this object" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:37.893078 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.893058    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw64r\" (UniqueName: \"kubernetes.io/projected/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-kube-api-access-cw64r\") pod \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\" (UID: \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\") "
Sep 16 00:54:37.893165 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.893145    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-extensions-socket-volume\") pod \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\" (UID: \"ab5a82e1-1390-4e10-aa6d-d84ae77dcd16\") "
Sep 16 00:54:37.893482 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.893457    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" (UID: "ab5a82e1-1390-4e10-aa6d-d84ae77dcd16"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:54:37.895245 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.895226    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-kube-api-access-cw64r" (OuterVolumeSpecName: "kube-api-access-cw64r") pod "ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" (UID: "ab5a82e1-1390-4e10-aa6d-d84ae77dcd16"). InnerVolumeSpecName "kube-api-access-cw64r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:54:37.968451 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.968421    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:37.974692 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.974676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:37.994136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.994110    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-extensions-socket-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:54:37.994246 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:37.994142    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw64r\" (UniqueName: \"kubernetes.io/projected/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16-kube-api-access-cw64r\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:54:38.263183 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.263122    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"]
Sep 16 00:54:38.265679 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:54:38.265654    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057d261b_74f4_4e5f_97ca_b89abff9c4e5.slice/crio-e8d53e25012fcc562082ac24e0bdfdb1cf8ec4cbb20c4aadec291acb52da78ef WatchSource:0}: Error finding container e8d53e25012fcc562082ac24e0bdfdb1cf8ec4cbb20c4aadec291acb52da78ef: Status 404 returned error can't find the container with id e8d53e25012fcc562082ac24e0bdfdb1cf8ec4cbb20c4aadec291acb52da78ef
Sep 16 00:54:38.269096 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.269065    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"]
Sep 16 00:54:38.271245 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:54:38.271221    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9468dff9_8f91_4873_9127_90317ab35382.slice/crio-e00f12466842527055013c2be59b818f2f13a56e5202f33684a8e2aa4210ce1d WatchSource:0}: Error finding container e00f12466842527055013c2be59b818f2f13a56e5202f33684a8e2aa4210ce1d: Status 404 returned error can't find the container with id e00f12466842527055013c2be59b818f2f13a56e5202f33684a8e2aa4210ce1d
Sep 16 00:54:38.424284 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.424250    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" containerID="3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0" exitCode=0
Sep 16 00:54:38.424417 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.424296    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:38.424417 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.424353    2573 scope.go:117] "RemoveContainer" containerID="3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0"
Sep 16 00:54:38.426097 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:38.426068    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" is forbidden: User \"system:node:ip-10-0-131-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-89.ec2.internal' and this object" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:38.426397 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.426373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx" event={"ID":"9468dff9-8f91-4873-9127-90317ab35382","Type":"ContainerStarted","Data":"4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23"}
Sep 16 00:54:38.426590 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.426424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx" event={"ID":"9468dff9-8f91-4873-9127-90317ab35382","Type":"ContainerStarted","Data":"e00f12466842527055013c2be59b818f2f13a56e5202f33684a8e2aa4210ce1d"}
Sep 16 00:54:38.426590 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.426456    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:38.428395 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.428365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm" event={"ID":"057d261b-74f4-4e5f-97ca-b89abff9c4e5","Type":"ContainerStarted","Data":"af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4"}
Sep 16 00:54:38.428486 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.428422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm" event={"ID":"057d261b-74f4-4e5f-97ca-b89abff9c4e5","Type":"ContainerStarted","Data":"e8d53e25012fcc562082ac24e0bdfdb1cf8ec4cbb20c4aadec291acb52da78ef"}
Sep 16 00:54:38.428590 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.428545    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:38.429007 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:38.428977    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" is forbidden: User \"system:node:ip-10-0-131-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-89.ec2.internal' and this object" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:38.434650 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.434565    2573 scope.go:117] "RemoveContainer" containerID="3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0"
Sep 16 00:54:38.434857 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:38.434837    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0\": container with ID starting with 3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0 not found: ID does not exist" containerID="3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0"
Sep 16 00:54:38.434930 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.434870    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0"} err="failed to get container status \"3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0\": rpc error: code = NotFound desc = could not find container \"3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0\": container with ID starting with 3121b46e08d9ff4c4c2d875b6ff9b8d4c0e74d8b77fea34e076cc799fb01a1c0 not found: ID does not exist"
Sep 16 00:54:38.451813 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:38.451783    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4lz7t\" is forbidden: User \"system:node:ip-10-0-131-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-89.ec2.internal' and this object" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4lz7t"
Sep 16 00:54:38.452451 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.452397    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx" podStartSLOduration=1.452377783 podStartE2EDuration="1.452377783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:54:38.449928184 +0000 UTC m=+664.152750379" watchObservedRunningTime="2026-09-16 00:54:38.452377783 +0000 UTC m=+664.155199977"
Sep 16 00:54:38.469868 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.469829    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm" podStartSLOduration=1.469819291 podStartE2EDuration="1.469819291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:54:38.467886136 +0000 UTC m=+664.170708331" watchObservedRunningTime="2026-09-16 00:54:38.469819291 +0000 UTC m=+664.172641485"
Sep 16 00:54:38.770952 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:38.770921    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5a82e1-1390-4e10-aa6d-d84ae77dcd16" path="/var/lib/kubelet/pods/ab5a82e1-1390-4e10-aa6d-d84ae77dcd16/volumes"
Sep 16 00:54:49.438618 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.438588    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:49.439075 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.438657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:54:49.504808 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.504773    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"]
Sep 16 00:54:49.504989 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.504952    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx" podUID="9468dff9-8f91-4873-9127-90317ab35382" containerName="manager" containerID="cri-o://4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23" gracePeriod=10
Sep 16 00:54:49.733969 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.733941    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"]
Sep 16 00:54:49.736779 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.736569    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:49.751346 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.751328    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:49.758764 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.758742    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"]
Sep 16 00:54:49.791592 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.791573    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9468dff9-8f91-4873-9127-90317ab35382-extensions-socket-volume\") pod \"9468dff9-8f91-4873-9127-90317ab35382\" (UID: \"9468dff9-8f91-4873-9127-90317ab35382\") "
Sep 16 00:54:49.791733 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.791718    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsrm6\" (UniqueName: \"kubernetes.io/projected/9468dff9-8f91-4873-9127-90317ab35382-kube-api-access-rsrm6\") pod \"9468dff9-8f91-4873-9127-90317ab35382\" (UID: \"9468dff9-8f91-4873-9127-90317ab35382\") "
Sep 16 00:54:49.791879 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.791862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mlnjw\" (UID: \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:49.791939 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.791906    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9468dff9-8f91-4873-9127-90317ab35382-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9468dff9-8f91-4873-9127-90317ab35382" (UID: "9468dff9-8f91-4873-9127-90317ab35382"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:54:49.791994 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.791977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd55s\" (UniqueName: \"kubernetes.io/projected/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-kube-api-access-nd55s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mlnjw\" (UID: \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:49.792065 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.792051    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9468dff9-8f91-4873-9127-90317ab35382-extensions-socket-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:54:49.793915 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.793890    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9468dff9-8f91-4873-9127-90317ab35382-kube-api-access-rsrm6" (OuterVolumeSpecName: "kube-api-access-rsrm6") pod "9468dff9-8f91-4873-9127-90317ab35382" (UID: "9468dff9-8f91-4873-9127-90317ab35382"). InnerVolumeSpecName "kube-api-access-rsrm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:54:49.892977 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.892948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd55s\" (UniqueName: \"kubernetes.io/projected/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-kube-api-access-nd55s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mlnjw\" (UID: \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:49.893148 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.893020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mlnjw\" (UID: \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:49.893148 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.893067    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrm6\" (UniqueName: \"kubernetes.io/projected/9468dff9-8f91-4873-9127-90317ab35382-kube-api-access-rsrm6\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:54:49.893347 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.893333    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mlnjw\" (UID: \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:49.904774 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:49.904756    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd55s\" (UniqueName: \"kubernetes.io/projected/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-kube-api-access-nd55s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mlnjw\" (UID: \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:50.058341 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.058265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:50.351085 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.351010    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"]
Sep 16 00:54:50.354444 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:54:50.354421    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6a2e87_2da9_4d7e_83ce_2ac266f77e5c.slice/crio-ab6d86c49511ef9cd60bb3ce98c88b12613249d6845cc5af327e53e92df45434 WatchSource:0}: Error finding container ab6d86c49511ef9cd60bb3ce98c88b12613249d6845cc5af327e53e92df45434: Status 404 returned error can't find the container with id ab6d86c49511ef9cd60bb3ce98c88b12613249d6845cc5af327e53e92df45434
Sep 16 00:54:50.493240 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.493205    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw" event={"ID":"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c","Type":"ContainerStarted","Data":"3ee23b580a8a3f50d915c49b24886db543c64904a3ccb65415d3c10ec04a0a77"}
Sep 16 00:54:50.493678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.493360    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:54:50.493678 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.493391    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw" event={"ID":"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c","Type":"ContainerStarted","Data":"ab6d86c49511ef9cd60bb3ce98c88b12613249d6845cc5af327e53e92df45434"}
Sep 16 00:54:50.494683 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.494660    2573 generic.go:356] "Generic (PLEG): container finished" podID="9468dff9-8f91-4873-9127-90317ab35382" containerID="4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23" exitCode=0
Sep 16 00:54:50.494756 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.494700    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"
Sep 16 00:54:50.494796 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.494702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx" event={"ID":"9468dff9-8f91-4873-9127-90317ab35382","Type":"ContainerDied","Data":"4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23"}
Sep 16 00:54:50.494854 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.494836    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx" event={"ID":"9468dff9-8f91-4873-9127-90317ab35382","Type":"ContainerDied","Data":"e00f12466842527055013c2be59b818f2f13a56e5202f33684a8e2aa4210ce1d"}
Sep 16 00:54:50.494906 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.494869    2573 scope.go:117] "RemoveContainer" containerID="4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23"
Sep 16 00:54:50.503509 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.503493    2573 scope.go:117] "RemoveContainer" containerID="4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23"
Sep 16 00:54:50.503806 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:54:50.503789    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23\": container with ID starting with 4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23 not found: ID does not exist" containerID="4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23"
Sep 16 00:54:50.503862 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.503815    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23"} err="failed to get container status \"4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23\": rpc error: code = NotFound desc = could not find container \"4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23\": container with ID starting with 4934dd8ce0125c664207aaa6d2a97571dec32772d67c72def900df0280b02b23 not found: ID does not exist"
Sep 16 00:54:50.517105 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.517064    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw" podStartSLOduration=1.517051972 podStartE2EDuration="1.517051972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:54:50.515726369 +0000 UTC m=+676.218548565" watchObservedRunningTime="2026-09-16 00:54:50.517051972 +0000 UTC m=+676.219874167"
Sep 16 00:54:50.537216 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.537195    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"]
Sep 16 00:54:50.546356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.546337    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-95ztx"]
Sep 16 00:54:50.770845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:54:50.770814    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9468dff9-8f91-4873-9127-90317ab35382" path="/var/lib/kubelet/pods/9468dff9-8f91-4873-9127-90317ab35382/volumes"
Sep 16 00:55:01.503998 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.503968    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 00:55:01.555258 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.555222    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"]
Sep 16 00:55:01.555487 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.555460    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm" podUID="057d261b-74f4-4e5f-97ca-b89abff9c4e5" containerName="manager" containerID="cri-o://af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4" gracePeriod=10
Sep 16 00:55:01.808921 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.808900    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:55:01.888536 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.888488    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/057d261b-74f4-4e5f-97ca-b89abff9c4e5-extensions-socket-volume\") pod \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\" (UID: \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\") "
Sep 16 00:55:01.888692 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.888648    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjhft\" (UniqueName: \"kubernetes.io/projected/057d261b-74f4-4e5f-97ca-b89abff9c4e5-kube-api-access-xjhft\") pod \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\" (UID: \"057d261b-74f4-4e5f-97ca-b89abff9c4e5\") "
Sep 16 00:55:01.888926 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.888908    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057d261b-74f4-4e5f-97ca-b89abff9c4e5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "057d261b-74f4-4e5f-97ca-b89abff9c4e5" (UID: "057d261b-74f4-4e5f-97ca-b89abff9c4e5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:55:01.890777 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.890758    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057d261b-74f4-4e5f-97ca-b89abff9c4e5-kube-api-access-xjhft" (OuterVolumeSpecName: "kube-api-access-xjhft") pod "057d261b-74f4-4e5f-97ca-b89abff9c4e5" (UID: "057d261b-74f4-4e5f-97ca-b89abff9c4e5"). InnerVolumeSpecName "kube-api-access-xjhft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:55:01.990050 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.990006    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjhft\" (UniqueName: \"kubernetes.io/projected/057d261b-74f4-4e5f-97ca-b89abff9c4e5-kube-api-access-xjhft\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:55:01.990050 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:01.990043    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/057d261b-74f4-4e5f-97ca-b89abff9c4e5-extensions-socket-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:55:02.555273 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.555242    2573 generic.go:356] "Generic (PLEG): container finished" podID="057d261b-74f4-4e5f-97ca-b89abff9c4e5" containerID="af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4" exitCode=0
Sep 16 00:55:02.555776 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.555301    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"
Sep 16 00:55:02.555776 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.555322    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm" event={"ID":"057d261b-74f4-4e5f-97ca-b89abff9c4e5","Type":"ContainerDied","Data":"af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4"}
Sep 16 00:55:02.555776 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.555376    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm" event={"ID":"057d261b-74f4-4e5f-97ca-b89abff9c4e5","Type":"ContainerDied","Data":"e8d53e25012fcc562082ac24e0bdfdb1cf8ec4cbb20c4aadec291acb52da78ef"}
Sep 16 00:55:02.555776 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.555399    2573 scope.go:117] "RemoveContainer" containerID="af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4"
Sep 16 00:55:02.566464 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.566447    2573 scope.go:117] "RemoveContainer" containerID="af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4"
Sep 16 00:55:02.566986 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:55:02.566957    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4\": container with ID starting with af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4 not found: ID does not exist" containerID="af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4"
Sep 16 00:55:02.567070 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.566983    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4"} err="failed to get container status \"af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4\": rpc error: code = NotFound desc = could not find container \"af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4\": container with ID starting with af07bb5f8b396ef9b7f51024abb3d8ba484ca9eb1e6245ecff37426504cc02f4 not found: ID does not exist"
Sep 16 00:55:02.577241 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.577201    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"]
Sep 16 00:55:02.584213 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.584192    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6pddm"]
Sep 16 00:55:02.771474 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:02.771444    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="057d261b-74f4-4e5f-97ca-b89abff9c4e5" path="/var/lib/kubelet/pods/057d261b-74f4-4e5f-97ca-b89abff9c4e5/volumes"
Sep 16 00:55:05.943157 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.943117    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"]
Sep 16 00:55:05.943550 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.943470    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057d261b-74f4-4e5f-97ca-b89abff9c4e5" containerName="manager"
Sep 16 00:55:05.943550 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.943482    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="057d261b-74f4-4e5f-97ca-b89abff9c4e5" containerName="manager"
Sep 16 00:55:05.943550 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.943503    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9468dff9-8f91-4873-9127-90317ab35382" containerName="manager"
Sep 16 00:55:05.943550 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.943508    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9468dff9-8f91-4873-9127-90317ab35382" containerName="manager"
Sep 16 00:55:05.943682 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.943593    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="057d261b-74f4-4e5f-97ca-b89abff9c4e5" containerName="manager"
Sep 16 00:55:05.943682 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.943606    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9468dff9-8f91-4873-9127-90317ab35382" containerName="manager"
Sep 16 00:55:05.947983 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.947964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:05.950329 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.950307    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-225b4\""
Sep 16 00:55:05.956951 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:05.956926    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"]
Sep 16 00:55:06.027776 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.027738    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.027960 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.027785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.027960 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.027811    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.027960 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.027854    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.028092 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.027950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.028092 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.027978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.028092 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.028013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.028092 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.028058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9baea0-1171-4f4e-a686-75ad536a98e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.028092 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.028077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59nb\" (UniqueName: \"kubernetes.io/projected/2f9baea0-1171-4f4e-a686-75ad536a98e7-kube-api-access-l59nb\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129418 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129418 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9baea0-1171-4f4e-a686-75ad536a98e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129684 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59nb\" (UniqueName: \"kubernetes.io/projected/2f9baea0-1171-4f4e-a686-75ad536a98e7-kube-api-access-l59nb\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129684 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129491    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129684 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129684 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129684 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129684 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.129684 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.130003 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129883    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.130003 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.129908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.130108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.130006    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.130108 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.130088    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.130345 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.130327    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9baea0-1171-4f4e-a686-75ad536a98e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.132057 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.132022    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.132175 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.132157    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.137201 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.137179    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9baea0-1171-4f4e-a686-75ad536a98e7-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.137311 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.137299    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59nb\" (UniqueName: \"kubernetes.io/projected/2f9baea0-1171-4f4e-a686-75ad536a98e7-kube-api-access-l59nb\") pod \"maas-default-gateway-openshift-default-5ccb968c66-kdnf6\" (UID: \"2f9baea0-1171-4f4e-a686-75ad536a98e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.258135 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.258108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:06.538439 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.538415    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"]
Sep 16 00:55:06.540856 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:55:06.540829    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9baea0_1171_4f4e_a686_75ad536a98e7.slice/crio-5b2dbffd7dd855949326431fb6f646a7a925c1daab161db99e7455b96c1aa4ff WatchSource:0}: Error finding container 5b2dbffd7dd855949326431fb6f646a7a925c1daab161db99e7455b96c1aa4ff: Status 404 returned error can't find the container with id 5b2dbffd7dd855949326431fb6f646a7a925c1daab161db99e7455b96c1aa4ff
Sep 16 00:55:06.543145 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.543101    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:55:06.543235 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.543195    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:55:06.543235 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.543228    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:55:06.576495 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:06.576468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" event={"ID":"2f9baea0-1171-4f4e-a686-75ad536a98e7","Type":"ContainerStarted","Data":"5b2dbffd7dd855949326431fb6f646a7a925c1daab161db99e7455b96c1aa4ff"}
Sep 16 00:55:07.582865 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:07.582831    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" event={"ID":"2f9baea0-1171-4f4e-a686-75ad536a98e7","Type":"ContainerStarted","Data":"26694694102cc9809f9f4828791e22179abd1c7dfefd4752a976c0b1529f8ac8"}
Sep 16 00:55:07.601230 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:07.601182    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" podStartSLOduration=2.601167345 podStartE2EDuration="2.601167345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:55:07.599372164 +0000 UTC m=+693.302194371" watchObservedRunningTime="2026-09-16 00:55:07.601167345 +0000 UTC m=+693.303989536"
Sep 16 00:55:08.259184 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:08.259154    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:08.263844 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:08.263819    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:08.588429 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:08.588348    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:08.589395 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:08.589377    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:55:21.379537 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.379500    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nwx8m"]
Sep 16 00:55:21.384114 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.384094    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m"
Sep 16 00:55:21.386321 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.386300    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rqnv6\""
Sep 16 00:55:21.390342 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.390320    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nwx8m"]
Sep 16 00:55:21.459773 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.459747    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86c52\" (UniqueName: \"kubernetes.io/projected/7e6d468b-dc39-4052-908d-9e1d08598f2c-kube-api-access-86c52\") pod \"authorino-7bcfd8f9cd-nwx8m\" (UID: \"7e6d468b-dc39-4052-908d-9e1d08598f2c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m"
Sep 16 00:55:21.560460 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.560429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86c52\" (UniqueName: \"kubernetes.io/projected/7e6d468b-dc39-4052-908d-9e1d08598f2c-kube-api-access-86c52\") pod \"authorino-7bcfd8f9cd-nwx8m\" (UID: \"7e6d468b-dc39-4052-908d-9e1d08598f2c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m"
Sep 16 00:55:21.567886 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.567865    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86c52\" (UniqueName: \"kubernetes.io/projected/7e6d468b-dc39-4052-908d-9e1d08598f2c-kube-api-access-86c52\") pod \"authorino-7bcfd8f9cd-nwx8m\" (UID: \"7e6d468b-dc39-4052-908d-9e1d08598f2c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m"
Sep 16 00:55:21.703980 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.703956    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m"
Sep 16 00:55:21.975367 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:21.975345    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nwx8m"]
Sep 16 00:55:21.977386 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:55:21.977363    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e6d468b_dc39_4052_908d_9e1d08598f2c.slice/crio-dff12f03f61f4d7da3e8b208d56140ceac89b363036879665029b1def0fc7491 WatchSource:0}: Error finding container dff12f03f61f4d7da3e8b208d56140ceac89b363036879665029b1def0fc7491: Status 404 returned error can't find the container with id dff12f03f61f4d7da3e8b208d56140ceac89b363036879665029b1def0fc7491
Sep 16 00:55:22.654788 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:22.654756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m" event={"ID":"7e6d468b-dc39-4052-908d-9e1d08598f2c","Type":"ContainerStarted","Data":"dff12f03f61f4d7da3e8b208d56140ceac89b363036879665029b1def0fc7491"}
Sep 16 00:55:24.666851 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:24.666819    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m" event={"ID":"7e6d468b-dc39-4052-908d-9e1d08598f2c","Type":"ContainerStarted","Data":"5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b"}
Sep 16 00:55:24.681336 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:24.681295    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m" podStartSLOduration=1.564170753 podStartE2EDuration="3.681279588s" totalImagesPullingTime="2.117108835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:55:24.679550539 +0000 UTC m=+710.382372735" watchObservedRunningTime="2026-09-16 00:55:24.681279588 +0000 UTC m=+710.384101782"
Sep 16 00:55:43.243794 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.243757    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nwx8m"]
Sep 16 00:55:43.244266 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.243938    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m" podUID="7e6d468b-dc39-4052-908d-9e1d08598f2c" containerName="authorino" containerID="cri-o://5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b" gracePeriod=30
Sep 16 00:55:43.496567 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.496502    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m"
Sep 16 00:55:43.640280 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.640246    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86c52\" (UniqueName: \"kubernetes.io/projected/7e6d468b-dc39-4052-908d-9e1d08598f2c-kube-api-access-86c52\") pod \"7e6d468b-dc39-4052-908d-9e1d08598f2c\" (UID: \"7e6d468b-dc39-4052-908d-9e1d08598f2c\") "
Sep 16 00:55:43.642444 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.642414    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6d468b-dc39-4052-908d-9e1d08598f2c-kube-api-access-86c52" (OuterVolumeSpecName: "kube-api-access-86c52") pod "7e6d468b-dc39-4052-908d-9e1d08598f2c" (UID: "7e6d468b-dc39-4052-908d-9e1d08598f2c"). InnerVolumeSpecName "kube-api-access-86c52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:55:43.741028 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.741004    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86c52\" (UniqueName: \"kubernetes.io/projected/7e6d468b-dc39-4052-908d-9e1d08598f2c-kube-api-access-86c52\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:55:43.760288 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.760224    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e6d468b-dc39-4052-908d-9e1d08598f2c" containerID="5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b" exitCode=0
Sep 16 00:55:43.760288 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.760255    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m"
Sep 16 00:55:43.760454 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.760304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m" event={"ID":"7e6d468b-dc39-4052-908d-9e1d08598f2c","Type":"ContainerDied","Data":"5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b"}
Sep 16 00:55:43.760454 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.760345    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-nwx8m" event={"ID":"7e6d468b-dc39-4052-908d-9e1d08598f2c","Type":"ContainerDied","Data":"dff12f03f61f4d7da3e8b208d56140ceac89b363036879665029b1def0fc7491"}
Sep 16 00:55:43.760454 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.760363    2573 scope.go:117] "RemoveContainer" containerID="5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b"
Sep 16 00:55:43.769256 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.769239    2573 scope.go:117] "RemoveContainer" containerID="5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b"
Sep 16 00:55:43.769509 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:55:43.769489    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b\": container with ID starting with 5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b not found: ID does not exist" containerID="5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b"
Sep 16 00:55:43.769597 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.769532    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b"} err="failed to get container status \"5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b\": rpc error: code = NotFound desc = could not find container \"5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b\": container with ID starting with 5eccf2313d506cd65fe8ba2f776ed8d06618851d9595b4e2a2b23d55cc0b5b2b not found: ID does not exist"
Sep 16 00:55:43.780121 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.780100    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nwx8m"]
Sep 16 00:55:43.781658 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:43.781641    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-nwx8m"]
Sep 16 00:55:44.770814 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:44.770776    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e6d468b-dc39-4052-908d-9e1d08598f2c" path="/var/lib/kubelet/pods/7e6d468b-dc39-4052-908d-9e1d08598f2c/volumes"
Sep 16 00:55:53.297740 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.297702    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-bkh77"]
Sep 16 00:55:53.298308 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.298247    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e6d468b-dc39-4052-908d-9e1d08598f2c" containerName="authorino"
Sep 16 00:55:53.298308 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.298267    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e6d468b-dc39-4052-908d-9e1d08598f2c" containerName="authorino"
Sep 16 00:55:53.298430 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.298351    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e6d468b-dc39-4052-908d-9e1d08598f2c" containerName="authorino"
Sep 16 00:55:53.302696 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.302673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-bkh77"
Sep 16 00:55:53.304712 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.304693    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rqnv6\""
Sep 16 00:55:53.307933 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.307914    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-bkh77"]
Sep 16 00:55:53.351365 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.351338    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-bkh77"]
Sep 16 00:55:53.351592 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:55:53.351573    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-f58gd], unattached volumes=[], failed to process volumes=[kube-api-access-f58gd]: context canceled" pod="kuadrant-system/authorino-557f876f69-bkh77" podUID="a07ec931-2324-4892-b12c-3d52bce9a8ab"
Sep 16 00:55:53.380356 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.380325    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tfr9t"]
Sep 16 00:55:53.383722 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.383702    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.385879 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.385861    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 00:55:53.390845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.390823    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tfr9t"]
Sep 16 00:55:53.415306 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.415286    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58gd\" (UniqueName: \"kubernetes.io/projected/a07ec931-2324-4892-b12c-3d52bce9a8ab-kube-api-access-f58gd\") pod \"authorino-557f876f69-bkh77\" (UID: \"a07ec931-2324-4892-b12c-3d52bce9a8ab\") " pod="kuadrant-system/authorino-557f876f69-bkh77"
Sep 16 00:55:53.516395 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.516369    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxdn\" (UniqueName: \"kubernetes.io/projected/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-kube-api-access-mrxdn\") pod \"authorino-6d4dfc649f-tfr9t\" (UID: \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\") " pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.516514 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.516429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58gd\" (UniqueName: \"kubernetes.io/projected/a07ec931-2324-4892-b12c-3d52bce9a8ab-kube-api-access-f58gd\") pod \"authorino-557f876f69-bkh77\" (UID: \"a07ec931-2324-4892-b12c-3d52bce9a8ab\") " pod="kuadrant-system/authorino-557f876f69-bkh77"
Sep 16 00:55:53.516514 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.516480    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-tls-cert\") pod \"authorino-6d4dfc649f-tfr9t\" (UID: \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\") " pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.523850 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.523829    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58gd\" (UniqueName: \"kubernetes.io/projected/a07ec931-2324-4892-b12c-3d52bce9a8ab-kube-api-access-f58gd\") pod \"authorino-557f876f69-bkh77\" (UID: \"a07ec931-2324-4892-b12c-3d52bce9a8ab\") " pod="kuadrant-system/authorino-557f876f69-bkh77"
Sep 16 00:55:53.559893 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.559824    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tfr9t"]
Sep 16 00:55:53.560117 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:55:53.560094    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mrxdn tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-tfr9t" podUID="3c0417c2-e2eb-4728-a4af-b3d0ba3c7067"
Sep 16 00:55:53.586261 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.586241    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fc77d8b54-lfqjn"]
Sep 16 00:55:53.589623 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.589601    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:55:53.599080 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.599059    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc77d8b54-lfqjn"]
Sep 16 00:55:53.617366 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.617341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-tls-cert\") pod \"authorino-6d4dfc649f-tfr9t\" (UID: \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\") " pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.617453 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.617431    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxdn\" (UniqueName: \"kubernetes.io/projected/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-kube-api-access-mrxdn\") pod \"authorino-6d4dfc649f-tfr9t\" (UID: \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\") " pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.619783 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.619761    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-tls-cert\") pod \"authorino-6d4dfc649f-tfr9t\" (UID: \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\") " pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.624429 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.624414    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxdn\" (UniqueName: \"kubernetes.io/projected/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-kube-api-access-mrxdn\") pod \"authorino-6d4dfc649f-tfr9t\" (UID: \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\") " pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.718263 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.718242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kglkc\" (UniqueName: \"kubernetes.io/projected/d485b3bb-567b-4e11-92d2-5da5ef0c45af-kube-api-access-kglkc\") pod \"authorino-5fc77d8b54-lfqjn\" (UID: \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\") " pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:55:53.718374 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.718282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d485b3bb-567b-4e11-92d2-5da5ef0c45af-tls-cert\") pod \"authorino-5fc77d8b54-lfqjn\" (UID: \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\") " pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:55:53.805244 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.805223    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.805339 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.805223    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-bkh77"
Sep 16 00:55:53.818765 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.818698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kglkc\" (UniqueName: \"kubernetes.io/projected/d485b3bb-567b-4e11-92d2-5da5ef0c45af-kube-api-access-kglkc\") pod \"authorino-5fc77d8b54-lfqjn\" (UID: \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\") " pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:55:53.818867 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.818779    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d485b3bb-567b-4e11-92d2-5da5ef0c45af-tls-cert\") pod \"authorino-5fc77d8b54-lfqjn\" (UID: \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\") " pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:55:53.821409 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.821391    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d485b3bb-567b-4e11-92d2-5da5ef0c45af-tls-cert\") pod \"authorino-5fc77d8b54-lfqjn\" (UID: \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\") " pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:55:53.826447 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.826430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kglkc\" (UniqueName: \"kubernetes.io/projected/d485b3bb-567b-4e11-92d2-5da5ef0c45af-kube-api-access-kglkc\") pod \"authorino-5fc77d8b54-lfqjn\" (UID: \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\") " pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:55:53.827332 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.827313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:53.843722 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.843704    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-bkh77"
Sep 16 00:55:53.918042 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.918014    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:55:53.919183 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.919167    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-tls-cert\") pod \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\" (UID: \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\") "
Sep 16 00:55:53.919261 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.919213    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f58gd\" (UniqueName: \"kubernetes.io/projected/a07ec931-2324-4892-b12c-3d52bce9a8ab-kube-api-access-f58gd\") pod \"a07ec931-2324-4892-b12c-3d52bce9a8ab\" (UID: \"a07ec931-2324-4892-b12c-3d52bce9a8ab\") "
Sep 16 00:55:53.919303 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.919282    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrxdn\" (UniqueName: \"kubernetes.io/projected/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-kube-api-access-mrxdn\") pod \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\" (UID: \"3c0417c2-e2eb-4728-a4af-b3d0ba3c7067\") "
Sep 16 00:55:53.921283 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.921263    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3c0417c2-e2eb-4728-a4af-b3d0ba3c7067" (UID: "3c0417c2-e2eb-4728-a4af-b3d0ba3c7067"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:55:53.921366 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.921340    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-kube-api-access-mrxdn" (OuterVolumeSpecName: "kube-api-access-mrxdn") pod "3c0417c2-e2eb-4728-a4af-b3d0ba3c7067" (UID: "3c0417c2-e2eb-4728-a4af-b3d0ba3c7067"). InnerVolumeSpecName "kube-api-access-mrxdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:55:53.921494 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:53.921475    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ec931-2324-4892-b12c-3d52bce9a8ab-kube-api-access-f58gd" (OuterVolumeSpecName: "kube-api-access-f58gd") pod "a07ec931-2324-4892-b12c-3d52bce9a8ab" (UID: "a07ec931-2324-4892-b12c-3d52bce9a8ab"). InnerVolumeSpecName "kube-api-access-f58gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:55:54.020778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.020748    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-tls-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:55:54.020778 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.020777    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58gd\" (UniqueName: \"kubernetes.io/projected/a07ec931-2324-4892-b12c-3d52bce9a8ab-kube-api-access-f58gd\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:55:54.020969 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.020791    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrxdn\" (UniqueName: \"kubernetes.io/projected/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067-kube-api-access-mrxdn\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:55:54.186619 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.186514    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc77d8b54-lfqjn"]
Sep 16 00:55:54.189356 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:55:54.189334    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd485b3bb_567b_4e11_92d2_5da5ef0c45af.slice/crio-e1f80be35cd3717d72a9165494963ec9fd2e93d331fc14c6c4fc1ec7b951a03a WatchSource:0}: Error finding container e1f80be35cd3717d72a9165494963ec9fd2e93d331fc14c6c4fc1ec7b951a03a: Status 404 returned error can't find the container with id e1f80be35cd3717d72a9165494963ec9fd2e93d331fc14c6c4fc1ec7b951a03a
Sep 16 00:55:54.811944 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.811883    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-bkh77"
Sep 16 00:55:54.811944 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.811906    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn" event={"ID":"d485b3bb-567b-4e11-92d2-5da5ef0c45af","Type":"ContainerStarted","Data":"8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56"}
Sep 16 00:55:54.812387 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.811952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn" event={"ID":"d485b3bb-567b-4e11-92d2-5da5ef0c45af","Type":"ContainerStarted","Data":"e1f80be35cd3717d72a9165494963ec9fd2e93d331fc14c6c4fc1ec7b951a03a"}
Sep 16 00:55:54.812387 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.812173    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-tfr9t"
Sep 16 00:55:54.828789 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.828733    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn" podStartSLOduration=1.33220455 podStartE2EDuration="1.828689586s" totalImagesPullingTime="496.485036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:55:54.826234872 +0000 UTC m=+740.529057076" watchObservedRunningTime="2026-09-16 00:55:54.828689586 +0000 UTC m=+740.531511789"
Sep 16 00:55:54.857701 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.857675    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tfr9t"]
Sep 16 00:55:54.867814 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.867789    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tfr9t"]
Sep 16 00:55:54.889531 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.889469    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-bkh77"]
Sep 16 00:55:54.898324 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:54.898298    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-bkh77"]
Sep 16 00:55:56.771540 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:56.771479    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c0417c2-e2eb-4728-a4af-b3d0ba3c7067" path="/var/lib/kubelet/pods/3c0417c2-e2eb-4728-a4af-b3d0ba3c7067/volumes"
Sep 16 00:55:56.771895 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:55:56.771745    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a07ec931-2324-4892-b12c-3d52bce9a8ab" path="/var/lib/kubelet/pods/a07ec931-2324-4892-b12c-3d52bce9a8ab/volumes"
Sep 16 00:56:36.851370 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:36.851330    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"]
Sep 16 00:56:36.856166 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:36.856132    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:36.858842 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:36.858811    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 00:56:36.859195 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:36.858846    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2xnr2\""
Sep 16 00:56:36.866963 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:36.866938    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"]
Sep 16 00:56:37.001870 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:37.001833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k58gh\" (UID: \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:37.002046 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:37.001982    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgc8f\" (UniqueName: \"kubernetes.io/projected/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-kube-api-access-vgc8f\") pod \"payload-pre-processing-59fbc47bf8-k58gh\" (UID: \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:37.102775 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:37.102685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgc8f\" (UniqueName: \"kubernetes.io/projected/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-kube-api-access-vgc8f\") pod \"payload-pre-processing-59fbc47bf8-k58gh\" (UID: \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:37.102945 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:37.102790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k58gh\" (UID: \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:37.103686 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:37.103665    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k58gh\" (UID: \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:37.110510 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:37.110484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgc8f\" (UniqueName: \"kubernetes.io/projected/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-kube-api-access-vgc8f\") pod \"payload-pre-processing-59fbc47bf8-k58gh\" (UID: \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:37.172223 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:37.172187    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:37.450265 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:37.450222    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"]
Sep 16 00:56:37.453015 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:56:37.452985    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5e8e68_9acf_46d6_bfff_4d4b2d73c043.slice/crio-4b30696f55f7fb7c561d7edb06cd11e68b7cad3a87250ae95a5bb5ded09a7f84 WatchSource:0}: Error finding container 4b30696f55f7fb7c561d7edb06cd11e68b7cad3a87250ae95a5bb5ded09a7f84: Status 404 returned error can't find the container with id 4b30696f55f7fb7c561d7edb06cd11e68b7cad3a87250ae95a5bb5ded09a7f84
Sep 16 00:56:38.016334 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:38.016300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh" event={"ID":"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043","Type":"ContainerStarted","Data":"4b30696f55f7fb7c561d7edb06cd11e68b7cad3a87250ae95a5bb5ded09a7f84"}
Sep 16 00:56:41.076426 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:41.076392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh" event={"ID":"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043","Type":"ContainerStarted","Data":"b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e"}
Sep 16 00:56:41.076849 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:41.076533    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:41.091468 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:41.091413    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh" podStartSLOduration=2.227522407 podStartE2EDuration="5.091392122s" totalImagesPullingTime="2.863869715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:56:41.090458894 +0000 UTC m=+786.793281088" watchObservedRunningTime="2026-09-16 00:56:41.091392122 +0000 UTC m=+786.794214318"
Sep 16 00:56:46.102131 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:46.102102    2573 generic.go:356] "Generic (PLEG): container finished" podID="2f9baea0-1171-4f4e-a686-75ad536a98e7" containerID="26694694102cc9809f9f4828791e22179abd1c7dfefd4752a976c0b1529f8ac8" exitCode=0
Sep 16 00:56:46.102437 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:46.102171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" event={"ID":"2f9baea0-1171-4f4e-a686-75ad536a98e7","Type":"ContainerDied","Data":"26694694102cc9809f9f4828791e22179abd1c7dfefd4752a976c0b1529f8ac8"}
Sep 16 00:56:46.102546 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:46.102514    2573 scope.go:117] "RemoveContainer" containerID="26694694102cc9809f9f4828791e22179abd1c7dfefd4752a976c0b1529f8ac8"
Sep 16 00:56:46.109459 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:46.103982    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:56:46.109459 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:46.104144    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:56:46.109459 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:46.104239    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:56:47.110502 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:47.110468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" event={"ID":"2f9baea0-1171-4f4e-a686-75ad536a98e7","Type":"ContainerStarted","Data":"493831f54051382785da877b1a929069b04ae467662832f8abe79bf41ed79513"}
Sep 16 00:56:47.258827 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:47.258787    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:56:47.259045 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:47.259019    2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-kdnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 00:56:47.259130 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:47.259097    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" podUID="2f9baea0-1171-4f4e-a686-75ad536a98e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 00:56:48.258975 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:48.258943    2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-kdnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 00:56:48.259374 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:48.259011    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" podUID="2f9baea0-1171-4f4e-a686-75ad536a98e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 00:56:49.259147 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:49.259113    2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-kdnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 00:56:49.259551 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:49.259168    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" podUID="2f9baea0-1171-4f4e-a686-75ad536a98e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 00:56:50.259466 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:50.259432    2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-kdnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 00:56:50.259870 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:50.259485    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" podUID="2f9baea0-1171-4f4e-a686-75ad536a98e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 00:56:51.258646 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:51.258607    2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-kdnf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 00:56:51.258831 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:51.258666    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6" podUID="2f9baea0-1171-4f4e-a686-75ad536a98e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 00:56:52.084312 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:52.084283    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:56:52.297615 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:52.297583    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:56:52.298104 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:52.298072    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:56:52.299245 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:56:52.299223    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-kdnf6"
Sep 16 00:57:08.476832 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.476750    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"]
Sep 16 00:57:08.480309 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.480285    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.483316 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.483295    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 00:57:08.483450 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.483336    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 00:57:08.483450 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.483352    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 00:57:08.483450 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.483359    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-spxkl\""
Sep 16 00:57:08.488210 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.488188    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"]
Sep 16 00:57:08.588204 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.588171    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.588341 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.588210    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwxk\" (UniqueName: \"kubernetes.io/projected/78b8a619-35cf-45de-adbd-8c2a2a40922b-kube-api-access-tzwxk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.588341 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.588310    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.588417 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.588343    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b8a619-35cf-45de-adbd-8c2a2a40922b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.588417 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.588364    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.588417 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.588380    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689303 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwxk\" (UniqueName: \"kubernetes.io/projected/78b8a619-35cf-45de-adbd-8c2a2a40922b-kube-api-access-tzwxk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689485 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689459    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b8a619-35cf-45de-adbd-8c2a2a40922b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689702 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689702 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689811 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689784    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689869 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689822    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.689971 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.689943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.691895 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.691876    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b8a619-35cf-45de-adbd-8c2a2a40922b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.692131 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.692116    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b8a619-35cf-45de-adbd-8c2a2a40922b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.697263 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.697244    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwxk\" (UniqueName: \"kubernetes.io/projected/78b8a619-35cf-45de-adbd-8c2a2a40922b-kube-api-access-tzwxk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9\" (UID: \"78b8a619-35cf-45de-adbd-8c2a2a40922b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:08.791047 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:08.790971    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:09.066559 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:09.066507    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"]
Sep 16 00:57:09.067961 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:57:09.067933    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b8a619_35cf_45de_adbd_8c2a2a40922b.slice/crio-b528a102cbf2655e66f23115afdf13ffe183c07433365f336ba0d6c23c142b38 WatchSource:0}: Error finding container b528a102cbf2655e66f23115afdf13ffe183c07433365f336ba0d6c23c142b38: Status 404 returned error can't find the container with id b528a102cbf2655e66f23115afdf13ffe183c07433365f336ba0d6c23c142b38
Sep 16 00:57:09.216980 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:09.216946    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9" event={"ID":"78b8a619-35cf-45de-adbd-8c2a2a40922b","Type":"ContainerStarted","Data":"b528a102cbf2655e66f23115afdf13ffe183c07433365f336ba0d6c23c142b38"}
Sep 16 00:57:11.229348 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:11.229308    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9" event={"ID":"78b8a619-35cf-45de-adbd-8c2a2a40922b","Type":"ContainerStarted","Data":"d745aef400925b6ab243f286758b14175487b69deba23bd97401c323981db5e8"}
Sep 16 00:57:11.246145 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:11.246093    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9" podStartSLOduration=1.956417676 podStartE2EDuration="3.246076533s" totalImagesPullingTime="1.289658857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:57:11.245284987 +0000 UTC m=+816.948107182" watchObservedRunningTime="2026-09-16 00:57:11.246076533 +0000 UTC m=+816.948898729"
Sep 16 00:57:18.791378 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:18.791348    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:18.791817 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:18.791669    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:18.803767 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:18.803741    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:19.282355 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:19.282324    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-phvs9"
Sep 16 00:57:49.781403 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.781369    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"]
Sep 16 00:57:49.786451 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.786427    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.788514 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.788494    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 00:57:49.793149 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.793122    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"]
Sep 16 00:57:49.838710 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.838681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjc6\" (UniqueName: \"kubernetes.io/projected/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-kube-api-access-rvjc6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.838846 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.838716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.838846 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.838756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.838921 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.838891    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.838961 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.838947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.838996 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.838977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940287 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940252    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940447 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940447 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940447 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940620 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjc6\" (UniqueName: \"kubernetes.io/projected/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-kube-api-access-rvjc6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940620 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940493    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940876 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940852    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940967 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940889    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.940967 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.940912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.942782 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.942766    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.943033 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.943016    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:49.948136 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:49.948116    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjc6\" (UniqueName: \"kubernetes.io/projected/a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6-kube-api-access-rvjc6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4\" (UID: \"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:50.097938 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:50.097854    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:57:50.380905 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:50.380875    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:57:50.382257 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:50.382159    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"]
Sep 16 00:57:50.437133 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:50.437101    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4" event={"ID":"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6","Type":"ContainerStarted","Data":"9ac20b6ef7a7fabb5c4b10543a93ac2b82c4a5038a4a0937bef1714946d973e8"}
Sep 16 00:57:51.444471 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:51.444438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4" event={"ID":"a2b68fe6-7e3a-4694-b8a7-b9be8a7485f6","Type":"ContainerStarted","Data":"fea9cc84dd47dab2043c697268a93b7dd25206b5720509866d4616ad9d5550cb"}
Sep 16 00:57:51.464180 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:57:51.464128    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4" podStartSLOduration=2.272443329 podStartE2EDuration="2.464106327s" totalImagesPullingTime="191.662998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:57:51.462897633 +0000 UTC m=+857.165719839" watchObservedRunningTime="2026-09-16 00:57:51.464106327 +0000 UTC m=+857.166928522"
Sep 16 00:58:00.098576 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:00.098542    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:58:00.098952 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:00.098607    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:58:00.111184 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:00.111159    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:58:00.501141 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:00.501116    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4mvm4"
Sep 16 00:58:39.383872 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.383839    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"]
Sep 16 00:58:39.387670 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.387652    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.390110 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.390079    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 00:58:39.401355 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.401333    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"]
Sep 16 00:58:39.464871 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.464842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.464871 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.464871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.465050 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.464900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.465050 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.464972    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tj8k\" (UniqueName: \"kubernetes.io/projected/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-kube-api-access-6tj8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.465050 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.464991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.465050 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.465007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.566489 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.566449    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.566489 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.566490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.566738 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.566558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.566738 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.566719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tj8k\" (UniqueName: \"kubernetes.io/projected/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-kube-api-access-6tj8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.566845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.566769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.566845 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.566801    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.566945 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.566849    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.566998 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.566953    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.567098 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.567082    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.568927 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.568908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.569255 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.569238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.573416 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.573398    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tj8k\" (UniqueName: \"kubernetes.io/projected/b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd-kube-api-access-6tj8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xczpb\" (UID: \"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.697455 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.697435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:39.969694 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:39.969471    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"]
Sep 16 00:58:39.973188 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:58:39.973159    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb40bf5ce_0f64_4c28_8b28_b41f8cb13cbd.slice/crio-89d276f5d2079875e067e2014f57f5cf3968631887d413175b50a88b69661b70 WatchSource:0}: Error finding container 89d276f5d2079875e067e2014f57f5cf3968631887d413175b50a88b69661b70: Status 404 returned error can't find the container with id 89d276f5d2079875e067e2014f57f5cf3968631887d413175b50a88b69661b70
Sep 16 00:58:40.691038 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:40.691005    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb" event={"ID":"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd","Type":"ContainerStarted","Data":"e1312d3fb1dd72937dec0131bcbec526fb299565aa9ff09b1a1941982914eab7"}
Sep 16 00:58:40.691038 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:40.691043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb" event={"ID":"b40bf5ce-0f64-4c28-8b28-b41f8cb13cbd","Type":"ContainerStarted","Data":"89d276f5d2079875e067e2014f57f5cf3968631887d413175b50a88b69661b70"}
Sep 16 00:58:40.708142 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:40.708100    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb" podStartSLOduration=1.520943293 podStartE2EDuration="1.708085006s" totalImagesPullingTime="187.141713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:58:40.705743944 +0000 UTC m=+906.408566138" watchObservedRunningTime="2026-09-16 00:58:40.708085006 +0000 UTC m=+906.410907200"
Sep 16 00:58:49.697855 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:49.697822    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:49.698306 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:49.698078    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:49.710902 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:49.710880    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:58:49.744076 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:58:49.744058    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xczpb"
Sep 16 00:59:40.416303 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.416264    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fc77d8b54-lfqjn"]
Sep 16 00:59:40.416894 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.416425    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn" podUID="d485b3bb-567b-4e11-92d2-5da5ef0c45af" containerName="authorino" containerID="cri-o://8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56" gracePeriod=30
Sep 16 00:59:40.663025 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.663001    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:59:40.680602 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.680487    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kglkc\" (UniqueName: \"kubernetes.io/projected/d485b3bb-567b-4e11-92d2-5da5ef0c45af-kube-api-access-kglkc\") pod \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\" (UID: \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\") "
Sep 16 00:59:40.680602 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.680542    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d485b3bb-567b-4e11-92d2-5da5ef0c45af-tls-cert\") pod \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\" (UID: \"d485b3bb-567b-4e11-92d2-5da5ef0c45af\") "
Sep 16 00:59:40.682792 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.682755    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d485b3bb-567b-4e11-92d2-5da5ef0c45af-kube-api-access-kglkc" (OuterVolumeSpecName: "kube-api-access-kglkc") pod "d485b3bb-567b-4e11-92d2-5da5ef0c45af" (UID: "d485b3bb-567b-4e11-92d2-5da5ef0c45af"). InnerVolumeSpecName "kube-api-access-kglkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:59:40.692667 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.692637    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d485b3bb-567b-4e11-92d2-5da5ef0c45af-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d485b3bb-567b-4e11-92d2-5da5ef0c45af" (UID: "d485b3bb-567b-4e11-92d2-5da5ef0c45af"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:59:40.781338 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.781317    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kglkc\" (UniqueName: \"kubernetes.io/projected/d485b3bb-567b-4e11-92d2-5da5ef0c45af-kube-api-access-kglkc\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:59:40.781338 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.781338    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d485b3bb-567b-4e11-92d2-5da5ef0c45af-tls-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:59:40.975372 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.975332    2573 generic.go:356] "Generic (PLEG): container finished" podID="d485b3bb-567b-4e11-92d2-5da5ef0c45af" containerID="8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56" exitCode=0
Sep 16 00:59:40.975372 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.975373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn" event={"ID":"d485b3bb-567b-4e11-92d2-5da5ef0c45af","Type":"ContainerDied","Data":"8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56"}
Sep 16 00:59:40.975632 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.975397    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn"
Sep 16 00:59:40.975632 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.975411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc77d8b54-lfqjn" event={"ID":"d485b3bb-567b-4e11-92d2-5da5ef0c45af","Type":"ContainerDied","Data":"e1f80be35cd3717d72a9165494963ec9fd2e93d331fc14c6c4fc1ec7b951a03a"}
Sep 16 00:59:40.975632 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.975432    2573 scope.go:117] "RemoveContainer" containerID="8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56"
Sep 16 00:59:40.986365 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.986344    2573 scope.go:117] "RemoveContainer" containerID="8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56"
Sep 16 00:59:40.986680 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:59:40.986647    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56\": container with ID starting with 8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56 not found: ID does not exist" containerID="8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56"
Sep 16 00:59:40.986771 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.986689    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56"} err="failed to get container status \"8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56\": rpc error: code = NotFound desc = could not find container \"8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56\": container with ID starting with 8d75bd43edef3db7a5e2dca41fa3043ad00a8fde85688ba51fd781339ddc5b56 not found: ID does not exist"
Sep 16 00:59:40.998999 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.998975    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"]
Sep 16 00:59:40.999145 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:40.999128    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh" podUID="ca5e8e68-9acf-46d6-bfff-4d4b2d73c043" containerName="payload-pre-processing" containerID="cri-o://b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e" gracePeriod=30
Sep 16 00:59:41.014643 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.014614    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fc77d8b54-lfqjn"]
Sep 16 00:59:41.026246 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.026222    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fc77d8b54-lfqjn"]
Sep 16 00:59:41.247327 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.247303    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:59:41.285743 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.285714    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgc8f\" (UniqueName: \"kubernetes.io/projected/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-kube-api-access-vgc8f\") pod \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\" (UID: \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\") "
Sep 16 00:59:41.285909 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.285856    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-plugins-config-volume\") pod \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\" (UID: \"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043\") "
Sep 16 00:59:41.286315 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.286273    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca5e8e68-9acf-46d6-bfff-4d4b2d73c043" (UID: "ca5e8e68-9acf-46d6-bfff-4d4b2d73c043"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:59:41.288084 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.288059    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-kube-api-access-vgc8f" (OuterVolumeSpecName: "kube-api-access-vgc8f") pod "ca5e8e68-9acf-46d6-bfff-4d4b2d73c043" (UID: "ca5e8e68-9acf-46d6-bfff-4d4b2d73c043"). InnerVolumeSpecName "kube-api-access-vgc8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:59:41.387590 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.387555    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:59:41.387590 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.387589    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgc8f\" (UniqueName: \"kubernetes.io/projected/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043-kube-api-access-vgc8f\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 00:59:41.984561 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.984507    2573 generic.go:356] "Generic (PLEG): container finished" podID="ca5e8e68-9acf-46d6-bfff-4d4b2d73c043" containerID="b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e" exitCode=0
Sep 16 00:59:41.984987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.984562    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh" event={"ID":"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043","Type":"ContainerDied","Data":"b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e"}
Sep 16 00:59:41.984987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.984579    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"
Sep 16 00:59:41.984987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.984635    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh" event={"ID":"ca5e8e68-9acf-46d6-bfff-4d4b2d73c043","Type":"ContainerDied","Data":"4b30696f55f7fb7c561d7edb06cd11e68b7cad3a87250ae95a5bb5ded09a7f84"}
Sep 16 00:59:41.984987 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.984673    2573 scope.go:117] "RemoveContainer" containerID="b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e"
Sep 16 00:59:41.994255 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.994234    2573 scope.go:117] "RemoveContainer" containerID="b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e"
Sep 16 00:59:41.994572 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:59:41.994542    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e\": container with ID starting with b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e not found: ID does not exist" containerID="b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e"
Sep 16 00:59:41.994664 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:41.994583    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e"} err="failed to get container status \"b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e\": rpc error: code = NotFound desc = could not find container \"b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e\": container with ID starting with b12414ab1b038cddef070ade750713ebf48fd185c1562ae69c6f27eee17f535e not found: ID does not exist"
Sep 16 00:59:41.999317 ip-10-0-131-89 kubenswrapper[2573]: E0916 00:59:41.999298    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5e8e68_9acf_46d6_bfff_4d4b2d73c043.slice/crio-4b30696f55f7fb7c561d7edb06cd11e68b7cad3a87250ae95a5bb5ded09a7f84\": RecentStats: unable to find data in memory cache]"
Sep 16 00:59:42.007167 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:42.007147    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"]
Sep 16 00:59:42.012499 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:42.012477    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k58gh"]
Sep 16 00:59:42.771999 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:42.771972    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca5e8e68-9acf-46d6-bfff-4d4b2d73c043" path="/var/lib/kubelet/pods/ca5e8e68-9acf-46d6-bfff-4d4b2d73c043/volumes"
Sep 16 00:59:42.772320 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:42.772308    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d485b3bb-567b-4e11-92d2-5da5ef0c45af" path="/var/lib/kubelet/pods/d485b3bb-567b-4e11-92d2-5da5ef0c45af/volumes"
Sep 16 00:59:55.124982 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.124935    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6d77c65979-pkw9c"]
Sep 16 00:59:55.125441 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.125410    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d485b3bb-567b-4e11-92d2-5da5ef0c45af" containerName="authorino"
Sep 16 00:59:55.125441 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.125431    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d485b3bb-567b-4e11-92d2-5da5ef0c45af" containerName="authorino"
Sep 16 00:59:55.125441 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.125443    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5e8e68-9acf-46d6-bfff-4d4b2d73c043" containerName="payload-pre-processing"
Sep 16 00:59:55.125441 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.125448    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5e8e68-9acf-46d6-bfff-4d4b2d73c043" containerName="payload-pre-processing"
Sep 16 00:59:55.125675 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.125546    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d485b3bb-567b-4e11-92d2-5da5ef0c45af" containerName="authorino"
Sep 16 00:59:55.125675 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.125563    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5e8e68-9acf-46d6-bfff-4d4b2d73c043" containerName="payload-pre-processing"
Sep 16 00:59:55.129828 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.129813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 00:59:55.131993 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.131972    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-7gkj5\""
Sep 16 00:59:55.132103 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.131972    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 16 00:59:55.135222 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.135203    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d77c65979-pkw9c"]
Sep 16 00:59:55.206343 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.206319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9bg\" (UniqueName: \"kubernetes.io/projected/c801dc83-43f0-4fb8-9cb9-65c36829fb4d-kube-api-access-lp9bg\") pod \"ai-gateway-controller-6d77c65979-pkw9c\" (UID: \"c801dc83-43f0-4fb8-9cb9-65c36829fb4d\") " pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 00:59:55.307497 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.307470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9bg\" (UniqueName: \"kubernetes.io/projected/c801dc83-43f0-4fb8-9cb9-65c36829fb4d-kube-api-access-lp9bg\") pod \"ai-gateway-controller-6d77c65979-pkw9c\" (UID: \"c801dc83-43f0-4fb8-9cb9-65c36829fb4d\") " pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 00:59:55.315131 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.315111    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9bg\" (UniqueName: \"kubernetes.io/projected/c801dc83-43f0-4fb8-9cb9-65c36829fb4d-kube-api-access-lp9bg\") pod \"ai-gateway-controller-6d77c65979-pkw9c\" (UID: \"c801dc83-43f0-4fb8-9cb9-65c36829fb4d\") " pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 00:59:55.441446 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.441379    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 00:59:55.711209 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:55.711185    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d77c65979-pkw9c"]
Sep 16 00:59:55.713701 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:59:55.713672    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc801dc83_43f0_4fb8_9cb9_65c36829fb4d.slice/crio-6d2be21ab273e6fd90a5815b3eef88427b2d0028239dfecdd4d30d006c7dbda9 WatchSource:0}: Error finding container 6d2be21ab273e6fd90a5815b3eef88427b2d0028239dfecdd4d30d006c7dbda9: Status 404 returned error can't find the container with id 6d2be21ab273e6fd90a5815b3eef88427b2d0028239dfecdd4d30d006c7dbda9
Sep 16 00:59:56.052161 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:56.052055    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c" event={"ID":"c801dc83-43f0-4fb8-9cb9-65c36829fb4d","Type":"ContainerStarted","Data":"6d2be21ab273e6fd90a5815b3eef88427b2d0028239dfecdd4d30d006c7dbda9"}
Sep 16 00:59:58.065761 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.065718    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c" event={"ID":"c801dc83-43f0-4fb8-9cb9-65c36829fb4d","Type":"ContainerStarted","Data":"8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f"}
Sep 16 00:59:58.066122 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.065924    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 00:59:58.081725 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.081640    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c" podStartSLOduration=0.971272458 podStartE2EDuration="3.081624879s" totalImagesPullingTime="2.110352421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:59:58.079676611 +0000 UTC m=+983.782498807" watchObservedRunningTime="2026-09-16 00:59:58.081624879 +0000 UTC m=+983.784447074"
Sep 16 00:59:58.239170 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.239140    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-87r96"]
Sep 16 00:59:58.242712 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.242690    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 00:59:58.244821 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.244789    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 00:59:58.244938 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.244789    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2xnr2\""
Sep 16 00:59:58.256581 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.256557    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-87r96"]
Sep 16 00:59:58.267027 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.267004    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"]
Sep 16 00:59:58.270832 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.270814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 00:59:58.281229 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.281209    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"]
Sep 16 00:59:58.338430 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.338353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3056300-83f7-42c9-97ac-82fff85a22b8-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-87r96\" (UID: \"b3056300-83f7-42c9-97ac-82fff85a22b8\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 00:59:58.338430 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.338387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569dl\" (UniqueName: \"kubernetes.io/projected/b3056300-83f7-42c9-97ac-82fff85a22b8-kube-api-access-569dl\") pod \"payload-processing-5d7dbbd56b-87r96\" (UID: \"b3056300-83f7-42c9-97ac-82fff85a22b8\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 00:59:58.338430 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.338410    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvqq\" (UniqueName: \"kubernetes.io/projected/bb2333dd-f5e0-4c73-af19-22ae153024c8-kube-api-access-gxvqq\") pod \"payload-pre-processing-6cd864bc95-gr57w\" (UID: \"bb2333dd-f5e0-4c73-af19-22ae153024c8\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 00:59:58.338677 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.338566    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2333dd-f5e0-4c73-af19-22ae153024c8-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-gr57w\" (UID: \"bb2333dd-f5e0-4c73-af19-22ae153024c8\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 00:59:58.439972 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.439939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3056300-83f7-42c9-97ac-82fff85a22b8-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-87r96\" (UID: \"b3056300-83f7-42c9-97ac-82fff85a22b8\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 00:59:58.439972 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.439977    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569dl\" (UniqueName: \"kubernetes.io/projected/b3056300-83f7-42c9-97ac-82fff85a22b8-kube-api-access-569dl\") pod \"payload-processing-5d7dbbd56b-87r96\" (UID: \"b3056300-83f7-42c9-97ac-82fff85a22b8\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 00:59:58.440219 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.439999    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvqq\" (UniqueName: \"kubernetes.io/projected/bb2333dd-f5e0-4c73-af19-22ae153024c8-kube-api-access-gxvqq\") pod \"payload-pre-processing-6cd864bc95-gr57w\" (UID: \"bb2333dd-f5e0-4c73-af19-22ae153024c8\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 00:59:58.440219 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.440077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2333dd-f5e0-4c73-af19-22ae153024c8-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-gr57w\" (UID: \"bb2333dd-f5e0-4c73-af19-22ae153024c8\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 00:59:58.440914 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.440889    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3056300-83f7-42c9-97ac-82fff85a22b8-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-87r96\" (UID: \"b3056300-83f7-42c9-97ac-82fff85a22b8\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 00:59:58.441032 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.440924    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2333dd-f5e0-4c73-af19-22ae153024c8-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-gr57w\" (UID: \"bb2333dd-f5e0-4c73-af19-22ae153024c8\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 00:59:58.448115 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.448094    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvqq\" (UniqueName: \"kubernetes.io/projected/bb2333dd-f5e0-4c73-af19-22ae153024c8-kube-api-access-gxvqq\") pod \"payload-pre-processing-6cd864bc95-gr57w\" (UID: \"bb2333dd-f5e0-4c73-af19-22ae153024c8\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 00:59:58.448203 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.448182    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-569dl\" (UniqueName: \"kubernetes.io/projected/b3056300-83f7-42c9-97ac-82fff85a22b8-kube-api-access-569dl\") pod \"payload-processing-5d7dbbd56b-87r96\" (UID: \"b3056300-83f7-42c9-97ac-82fff85a22b8\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 00:59:58.552679 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.552648    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 00:59:58.581588 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.581562    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 00:59:58.826154 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.826134    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-87r96"]
Sep 16 00:59:58.828278 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:59:58.828255    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3056300_83f7_42c9_97ac_82fff85a22b8.slice/crio-9167e60ac8ae5a1a7865761220e5b62e98e643644686b62a39ef93ff676d9380 WatchSource:0}: Error finding container 9167e60ac8ae5a1a7865761220e5b62e98e643644686b62a39ef93ff676d9380: Status 404 returned error can't find the container with id 9167e60ac8ae5a1a7865761220e5b62e98e643644686b62a39ef93ff676d9380
Sep 16 00:59:58.861750 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:58.861694    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"]
Sep 16 00:59:58.864772 ip-10-0-131-89 kubenswrapper[2573]: W0916 00:59:58.864746    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2333dd_f5e0_4c73_af19_22ae153024c8.slice/crio-fe9bb5811477be4f3b4edede10706b16e7b2bd70f8c5572d9f406b3be7366a67 WatchSource:0}: Error finding container fe9bb5811477be4f3b4edede10706b16e7b2bd70f8c5572d9f406b3be7366a67: Status 404 returned error can't find the container with id fe9bb5811477be4f3b4edede10706b16e7b2bd70f8c5572d9f406b3be7366a67
Sep 16 00:59:59.071536 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:59.071496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" event={"ID":"bb2333dd-f5e0-4c73-af19-22ae153024c8","Type":"ContainerStarted","Data":"fe9bb5811477be4f3b4edede10706b16e7b2bd70f8c5572d9f406b3be7366a67"}
Sep 16 00:59:59.072744 ip-10-0-131-89 kubenswrapper[2573]: I0916 00:59:59.072722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" event={"ID":"b3056300-83f7-42c9-97ac-82fff85a22b8","Type":"ContainerStarted","Data":"9167e60ac8ae5a1a7865761220e5b62e98e643644686b62a39ef93ff676d9380"}
Sep 16 01:00:00.135438 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:00.135414    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj"]
Sep 16 01:00:00.140343 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:00.140328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj"
Sep 16 01:00:00.142624 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:00.142603    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:00:00.144435 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:00.144398    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj"]
Sep 16 01:00:00.171261 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:00.171230    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj"
Sep 16 01:00:00.444974 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:00.444944    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj"]
Sep 16 01:00:00.447202 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:00:00.447176    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dcd791_4b8d_49a6_bf37_4453e193cae0.slice/crio-3e06b5344421fabb1294ed6a6bb9dab742e31ae1fb4ccb0a0ccd5393232d92ae WatchSource:0}: Error finding container 3e06b5344421fabb1294ed6a6bb9dab742e31ae1fb4ccb0a0ccd5393232d92ae: Status 404 returned error can't find the container with id 3e06b5344421fabb1294ed6a6bb9dab742e31ae1fb4ccb0a0ccd5393232d92ae
Sep 16 01:00:01.084142 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:01.084108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" event={"ID":"bb2333dd-f5e0-4c73-af19-22ae153024c8","Type":"ContainerStarted","Data":"26e34fa3e581ae9f623035c277dec43d470a90125fa2611d89451539581b057f"}
Sep 16 01:00:01.084342 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:01.084193    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 01:00:01.085410 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:01.085389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj" event={"ID":"35dcd791-4b8d-49a6-bf37-4453e193cae0","Type":"ContainerStarted","Data":"3e06b5344421fabb1294ed6a6bb9dab742e31ae1fb4ccb0a0ccd5393232d92ae"}
Sep 16 01:00:01.086877 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:01.086852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" event={"ID":"b3056300-83f7-42c9-97ac-82fff85a22b8","Type":"ContainerStarted","Data":"1595b6d6e3ad3cb59646433d4a99565c17509047847ae8583ee6b075df13722c"}
Sep 16 01:00:01.086941 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:01.086917    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 01:00:01.099360 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:01.099318    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" podStartSLOduration=1.860146987 podStartE2EDuration="3.099304031s" totalImagesPullingTime="1.239157044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:01.098166384 +0000 UTC m=+986.800988580" watchObservedRunningTime="2026-09-16 01:00:01.099304031 +0000 UTC m=+986.802126233"
Sep 16 01:00:01.112798 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:01.112758    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" podStartSLOduration=1.83970293 podStartE2EDuration="3.112743682s" totalImagesPullingTime="1.273040752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:01.111469628 +0000 UTC m=+986.814291824" watchObservedRunningTime="2026-09-16 01:00:01.112743682 +0000 UTC m=+986.815565888"
Sep 16 01:00:04.111784 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:04.111754    2573 generic.go:356] "Generic (PLEG): container finished" podID="35dcd791-4b8d-49a6-bf37-4453e193cae0" containerID="3e16c78cd7ad3435f9b71017522771c6686004e228ffec10e042729bd3ce5f13" exitCode=0
Sep 16 01:00:04.112128 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:04.111844    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj" event={"ID":"35dcd791-4b8d-49a6-bf37-4453e193cae0","Type":"ContainerDied","Data":"3e16c78cd7ad3435f9b71017522771c6686004e228ffec10e042729bd3ce5f13"}
Sep 16 01:00:05.254654 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:05.254633    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj"
Sep 16 01:00:06.123133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:06.123104    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj"
Sep 16 01:00:06.123336 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:06.123149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-6snkj" event={"ID":"35dcd791-4b8d-49a6-bf37-4453e193cae0","Type":"ContainerDied","Data":"3e06b5344421fabb1294ed6a6bb9dab742e31ae1fb4ccb0a0ccd5393232d92ae"}
Sep 16 01:00:06.123336 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:06.123198    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e06b5344421fabb1294ed6a6bb9dab742e31ae1fb4ccb0a0ccd5393232d92ae"
Sep 16 01:00:09.075289 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.075249    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 01:00:09.254307 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.254277    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-689bfb855b-xbd27"]
Sep 16 01:00:09.254677 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.254665    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35dcd791-4b8d-49a6-bf37-4453e193cae0" containerName="cleanup"
Sep 16 01:00:09.254720 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.254680    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="35dcd791-4b8d-49a6-bf37-4453e193cae0" containerName="cleanup"
Sep 16 01:00:09.254766 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.254757    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="35dcd791-4b8d-49a6-bf37-4453e193cae0" containerName="cleanup"
Sep 16 01:00:09.257755 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.257738    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27"
Sep 16 01:00:09.265691 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.265671    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-689bfb855b-xbd27"]
Sep 16 01:00:09.342383 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.342323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nx2\" (UniqueName: \"kubernetes.io/projected/6641e296-c644-4a27-ae3c-b2bc4df877b0-kube-api-access-49nx2\") pod \"ai-gateway-controller-689bfb855b-xbd27\" (UID: \"6641e296-c644-4a27-ae3c-b2bc4df877b0\") " pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27"
Sep 16 01:00:09.443003 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.442980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nx2\" (UniqueName: \"kubernetes.io/projected/6641e296-c644-4a27-ae3c-b2bc4df877b0-kube-api-access-49nx2\") pod \"ai-gateway-controller-689bfb855b-xbd27\" (UID: \"6641e296-c644-4a27-ae3c-b2bc4df877b0\") " pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27"
Sep 16 01:00:09.450916 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.450898    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nx2\" (UniqueName: \"kubernetes.io/projected/6641e296-c644-4a27-ae3c-b2bc4df877b0-kube-api-access-49nx2\") pod \"ai-gateway-controller-689bfb855b-xbd27\" (UID: \"6641e296-c644-4a27-ae3c-b2bc4df877b0\") " pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27"
Sep 16 01:00:09.569333 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.569312    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27"
Sep 16 01:00:09.840228 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:09.840204    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-689bfb855b-xbd27"]
Sep 16 01:00:09.842483 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:00:09.842455    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6641e296_c644_4a27_ae3c_b2bc4df877b0.slice/crio-8b3e9beb00ac6cfd76b35234c341eeae7a167eaca508a56f239c66a846782526 WatchSource:0}: Error finding container 8b3e9beb00ac6cfd76b35234c341eeae7a167eaca508a56f239c66a846782526: Status 404 returned error can't find the container with id 8b3e9beb00ac6cfd76b35234c341eeae7a167eaca508a56f239c66a846782526
Sep 16 01:00:10.145884 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:10.145852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27" event={"ID":"6641e296-c644-4a27-ae3c-b2bc4df877b0","Type":"ContainerStarted","Data":"8b3e9beb00ac6cfd76b35234c341eeae7a167eaca508a56f239c66a846782526"}
Sep 16 01:00:11.151991 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:11.151956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27" event={"ID":"6641e296-c644-4a27-ae3c-b2bc4df877b0","Type":"ContainerStarted","Data":"3308b75b03a1ca100c3c2f87a4c50ed9aa0870434de63d70815ab6d4219b02e7"}
Sep 16 01:00:11.152417 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:11.152080    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27"
Sep 16 01:00:11.170662 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:11.170617    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27" podStartSLOduration=1.39571691 podStartE2EDuration="2.170602546s" totalImagesPullingTime="774.885636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:11.168347015 +0000 UTC m=+996.871169211" watchObservedRunningTime="2026-09-16 01:00:11.170602546 +0000 UTC m=+996.873424741"
Sep 16 01:00:12.101120 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:12.101090    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96"
Sep 16 01:00:12.101402 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:12.101153    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w"
Sep 16 01:00:12.101675 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:12.101656    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:00:12.101776 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:12.101707    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:00:22.160744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:22.160713    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-689bfb855b-xbd27"
Sep 16 01:00:22.198635 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:22.198611    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6d77c65979-pkw9c"]
Sep 16 01:00:22.198847 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:22.198817    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c" podUID="c801dc83-43f0-4fb8-9cb9-65c36829fb4d" containerName="manager" containerID="cri-o://8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f" gracePeriod=30
Sep 16 01:00:22.444037 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:22.444014    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 01:00:22.463182 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:22.463158    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp9bg\" (UniqueName: \"kubernetes.io/projected/c801dc83-43f0-4fb8-9cb9-65c36829fb4d-kube-api-access-lp9bg\") pod \"c801dc83-43f0-4fb8-9cb9-65c36829fb4d\" (UID: \"c801dc83-43f0-4fb8-9cb9-65c36829fb4d\") "
Sep 16 01:00:22.465475 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:22.465449    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c801dc83-43f0-4fb8-9cb9-65c36829fb4d-kube-api-access-lp9bg" (OuterVolumeSpecName: "kube-api-access-lp9bg") pod "c801dc83-43f0-4fb8-9cb9-65c36829fb4d" (UID: "c801dc83-43f0-4fb8-9cb9-65c36829fb4d"). InnerVolumeSpecName "kube-api-access-lp9bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:00:22.563958 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:22.563933    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9bg\" (UniqueName: \"kubernetes.io/projected/c801dc83-43f0-4fb8-9cb9-65c36829fb4d-kube-api-access-lp9bg\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:00:23.215917 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.215885    2573 generic.go:356] "Generic (PLEG): container finished" podID="c801dc83-43f0-4fb8-9cb9-65c36829fb4d" containerID="8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f" exitCode=0
Sep 16 01:00:23.216349 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.215928    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c"
Sep 16 01:00:23.216349 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.215953    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c" event={"ID":"c801dc83-43f0-4fb8-9cb9-65c36829fb4d","Type":"ContainerDied","Data":"8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f"}
Sep 16 01:00:23.216349 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.216005    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d77c65979-pkw9c" event={"ID":"c801dc83-43f0-4fb8-9cb9-65c36829fb4d","Type":"ContainerDied","Data":"6d2be21ab273e6fd90a5815b3eef88427b2d0028239dfecdd4d30d006c7dbda9"}
Sep 16 01:00:23.216349 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.216035    2573 scope.go:117] "RemoveContainer" containerID="8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f"
Sep 16 01:00:23.224507 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.224491    2573 scope.go:117] "RemoveContainer" containerID="8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f"
Sep 16 01:00:23.224763 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:00:23.224743    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f\": container with ID starting with 8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f not found: ID does not exist" containerID="8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f"
Sep 16 01:00:23.224856 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.224772    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f"} err="failed to get container status \"8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f\": rpc error: code = NotFound desc = could not find container \"8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f\": container with ID starting with 8b95d406b40be334862f4f6e2cf29ac3cb95d14da6b3908ce896bd62e1f79e4f not found: ID does not exist"
Sep 16 01:00:23.230922 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.230905    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6d77c65979-pkw9c"]
Sep 16 01:00:23.236282 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:23.236261    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6d77c65979-pkw9c"]
Sep 16 01:00:24.770193 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:24.770159    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c801dc83-43f0-4fb8-9cb9-65c36829fb4d" path="/var/lib/kubelet/pods/c801dc83-43f0-4fb8-9cb9-65c36829fb4d/volumes"
Sep 16 01:00:42.075514 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.075483    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"]
Sep 16 01:00:42.075910 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.075895    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c801dc83-43f0-4fb8-9cb9-65c36829fb4d" containerName="manager"
Sep 16 01:00:42.075910 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.075908    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c801dc83-43f0-4fb8-9cb9-65c36829fb4d" containerName="manager"
Sep 16 01:00:42.075986 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.075969    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c801dc83-43f0-4fb8-9cb9-65c36829fb4d" containerName="manager"
Sep 16 01:00:42.083237 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.083217    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.087484 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.087458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d58169-openshift-default-dockercfg-xjf4x\""
Sep 16 01:00:42.089006 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.088971    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"]
Sep 16 01:00:42.129136 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.129108    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"]
Sep 16 01:00:42.132387 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.132365    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.135316 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.135292    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-96c840-openshift-default-dockercfg-d5sjq\""
Sep 16 01:00:42.145955 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.145931    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"]
Sep 16 01:00:42.196160 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.196130    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"]
Sep 16 01:00:42.199686 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.199664    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.202163 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.202140    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2534e3-openshift-default-dockercfg-pchcs\""
Sep 16 01:00:42.209895 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.209871    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"]
Sep 16 01:00:42.233573 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233543    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-envoy\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.233734 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233597    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-certs\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.233734 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qtrd\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-kube-api-access-6qtrd\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.233734 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-token\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.233903 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c81bd-f337-446c-9bce-977dd0fb592a-istiod-ca-cert\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.233903 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-data\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.233903 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-envoy\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.234050 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-data\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.234050 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233954    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-credential-socket\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.234050 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.233981    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-podinfo\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.234050 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.234007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-token\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.234211 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.234063    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-socket\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.234211 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.234093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-podinfo\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.234211 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.234128    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-certs\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.234211 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.234154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istiod-ca-cert\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.234211 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.234198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-credential-socket\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.234389 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.234248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-socket\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.234389 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.234274    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ppwv\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-kube-api-access-8ppwv\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.335628 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335546    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-credential-socket\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.335628 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335580    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-token\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.335628 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-podinfo\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.335628 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-token\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.335963 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-certs\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.335963 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-podinfo\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.335963 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335750    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-socket\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.335963 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-podinfo\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.335963 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-certs\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.335963 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istiod-ca-cert\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.335963 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.335962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-credential-socket\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-socket\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336062    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ppwv\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-kube-api-access-8ppwv\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-envoy\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336149    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87589a4b-1114-4d80-91b3-34f5270b9b6d-istiod-ca-cert\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336156    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-socket\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-certs\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336245    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-certs\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qtrd\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-kube-api-access-6qtrd\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.336312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336310    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-data\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.336830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-credential-socket\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.336830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336398    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-token\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.336830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c81bd-f337-446c-9bce-977dd0fb592a-istiod-ca-cert\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.336830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336469    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-socket\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.336830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336482    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzhk\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-kube-api-access-bxzhk\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.336830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336678    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-credential-socket\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.337157 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-certs\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.337157 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.336938    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istiod-ca-cert\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.337319 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-data\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.337466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-envoy\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.337611 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337488    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c81bd-f337-446c-9bce-977dd0fb592a-istiod-ca-cert\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.337611 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337495    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-data\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.337611 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-envoy\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.337611 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337308    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-credential-socket\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.337611 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337561    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-socket\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.337611 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-data\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.337949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.337874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-data\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.338882 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.338857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-podinfo\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.338967 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.338926    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-podinfo\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.339317 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.339299    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-envoy\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.339742 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.339727    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-envoy\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.343351 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.343322    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-token\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.343960 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.343943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ppwv\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-kube-api-access-8ppwv\") pod \"e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") " pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.344334 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.344317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qtrd\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-kube-api-access-6qtrd\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.344673 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.344657    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-token\") pod \"e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") " pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.397395 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.397361    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:42.439105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.438861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87589a4b-1114-4d80-91b3-34f5270b9b6d-istiod-ca-cert\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.438908    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-data\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.438937    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-credential-socket\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.438972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzhk\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-kube-api-access-bxzhk\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.439019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-envoy\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.439059    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-socket\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.439094    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-token\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439624 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.439134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-certs\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439624 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.439164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-podinfo\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439737 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.439674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-data\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439737 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.439703    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-socket\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.439841 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.439794    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-credential-socket\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.440083 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.440059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-certs\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.440306 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.440288    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87589a4b-1114-4d80-91b3-34f5270b9b6d-istiod-ca-cert\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.442196 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.442163    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-envoy\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.443152 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.443132    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-podinfo\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.446876 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.446853    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-token\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.446964 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.446930    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzhk\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-kube-api-access-bxzhk\") pod \"e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") " pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.447058 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.447043    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:42.512487 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.512334    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:42.679045 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.679018    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"]
Sep 16 01:00:42.679864 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:00:42.679838    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67c81bd_f337_446c_9bce_977dd0fb592a.slice/crio-81c190ca9bb6245ea60a256866a2330a677da4a9a291564fa495fa90b5816907 WatchSource:0}: Error finding container 81c190ca9bb6245ea60a256866a2330a677da4a9a291564fa495fa90b5816907: Status 404 returned error can't find the container with id 81c190ca9bb6245ea60a256866a2330a677da4a9a291564fa495fa90b5816907
Sep 16 01:00:42.682105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.682065    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:42.682214 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.682134    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:42.682214 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.682161    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:42.739116 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.739086    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"]
Sep 16 01:00:42.740051 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:00:42.740013    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06460b71_72f9_4fd3_93f8_d61f6ecd5be0.slice/crio-3de7da098a4f28d739bccdf8011a31f0e1e93f034effa03932c45372cfaa1107 WatchSource:0}: Error finding container 3de7da098a4f28d739bccdf8011a31f0e1e93f034effa03932c45372cfaa1107: Status 404 returned error can't find the container with id 3de7da098a4f28d739bccdf8011a31f0e1e93f034effa03932c45372cfaa1107
Sep 16 01:00:42.742101 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.742071    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:42.742174 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.742132    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:42.742174 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.742165    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:42.800448 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.800426    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"]
Sep 16 01:00:42.803253 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:00:42.803224    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87589a4b_1114_4d80_91b3_34f5270b9b6d.slice/crio-c7e5181cbfe2c7f26aaf4410f71e7d132bd057f7045d51be792284dba0a052c2 WatchSource:0}: Error finding container c7e5181cbfe2c7f26aaf4410f71e7d132bd057f7045d51be792284dba0a052c2: Status 404 returned error can't find the container with id c7e5181cbfe2c7f26aaf4410f71e7d132bd057f7045d51be792284dba0a052c2
Sep 16 01:00:42.806167 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.806129    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:42.806263 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.806208    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:42.806263 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:42.806250    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:43.319481 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.319448    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" event={"ID":"87589a4b-1114-4d80-91b3-34f5270b9b6d","Type":"ContainerStarted","Data":"6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504"}
Sep 16 01:00:43.320041 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.319502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" event={"ID":"87589a4b-1114-4d80-91b3-34f5270b9b6d","Type":"ContainerStarted","Data":"c7e5181cbfe2c7f26aaf4410f71e7d132bd057f7045d51be792284dba0a052c2"}
Sep 16 01:00:43.321230 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.321202    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" event={"ID":"c67c81bd-f337-446c-9bce-977dd0fb592a","Type":"ContainerStarted","Data":"58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2"}
Sep 16 01:00:43.321371 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.321248    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" event={"ID":"c67c81bd-f337-446c-9bce-977dd0fb592a","Type":"ContainerStarted","Data":"81c190ca9bb6245ea60a256866a2330a677da4a9a291564fa495fa90b5816907"}
Sep 16 01:00:43.322881 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.322859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" event={"ID":"06460b71-72f9-4fd3-93f8-d61f6ecd5be0","Type":"ContainerStarted","Data":"26554cfce7e1999be01a3336217853dd9578d2c4b2bd178618ad14513a11075d"}
Sep 16 01:00:43.322993 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.322896    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" event={"ID":"06460b71-72f9-4fd3-93f8-d61f6ecd5be0","Type":"ContainerStarted","Data":"3de7da098a4f28d739bccdf8011a31f0e1e93f034effa03932c45372cfaa1107"}
Sep 16 01:00:43.342207 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.342157    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podStartSLOduration=1.342141438 podStartE2EDuration="1.342141438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:43.340335786 +0000 UTC m=+1029.043158003" watchObservedRunningTime="2026-09-16 01:00:43.342141438 +0000 UTC m=+1029.044963666"
Sep 16 01:00:43.363103 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.363062    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" podStartSLOduration=1.3630478400000001 podStartE2EDuration="1.36304784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:43.361135105 +0000 UTC m=+1029.063957300" watchObservedRunningTime="2026-09-16 01:00:43.36304784 +0000 UTC m=+1029.065870035"
Sep 16 01:00:43.381644 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.381606    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" podStartSLOduration=1.381592471 podStartE2EDuration="1.381592471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:43.379407349 +0000 UTC m=+1029.082229580" watchObservedRunningTime="2026-09-16 01:00:43.381592471 +0000 UTC m=+1029.084414666"
Sep 16 01:00:43.397782 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.397760    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:43.402830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.402813    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:43.448298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.448278    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:43.453767 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.453748    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:43.513911 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.513879    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:43.519114 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:43.519087    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:44.328478 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:44.328358    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:44.328478 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:44.328416    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:44.328478 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:44.328444    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:44.329495 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:44.329469    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:00:44.329879 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:44.329857    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:00:44.330015 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:44.329969    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:00:48.265906 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.265872    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"]
Sep 16 01:00:48.270486 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.270468    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:48.272499 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.272482    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-2534e3-dockercfg-6blkl\""
Sep 16 01:00:48.272610 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.272508    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-2534e3\""
Sep 16 01:00:48.279299 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.279277    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"]
Sep 16 01:00:48.392327 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.392292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26927ad2-6d42-4ab7-bde1-6d363b14afcd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47\" (UID: \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:48.392495 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.392371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/26927ad2-6d42-4ab7-bde1-6d363b14afcd-kube-api-access-zv2s8\") pod \"payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47\" (UID: \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:48.493300 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.493265    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26927ad2-6d42-4ab7-bde1-6d363b14afcd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47\" (UID: \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:48.493473 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.493367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/26927ad2-6d42-4ab7-bde1-6d363b14afcd-kube-api-access-zv2s8\") pod \"payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47\" (UID: \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:48.494243 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.494211    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26927ad2-6d42-4ab7-bde1-6d363b14afcd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47\" (UID: \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:48.502796 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.502773    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/26927ad2-6d42-4ab7-bde1-6d363b14afcd-kube-api-access-zv2s8\") pod \"payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47\" (UID: \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:48.582113 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.582042    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:48.855298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:48.855277    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"]
Sep 16 01:00:48.857322 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:00:48.857296    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26927ad2_6d42_4ab7_bde1_6d363b14afcd.slice/crio-cea6bd38e9e13001a6c48fb1de5de9530ffddd1ad11613ec80128028f6bd97cb WatchSource:0}: Error finding container cea6bd38e9e13001a6c48fb1de5de9530ffddd1ad11613ec80128028f6bd97cb: Status 404 returned error can't find the container with id cea6bd38e9e13001a6c48fb1de5de9530ffddd1ad11613ec80128028f6bd97cb
Sep 16 01:00:49.124997 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.124915    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"]
Sep 16 01:00:49.128655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.128628    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:49.130851 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.130823    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d58169\""
Sep 16 01:00:49.130976 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.130958    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d58169-dockercfg-294sd\""
Sep 16 01:00:49.139561 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.139512    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"]
Sep 16 01:00:49.151708 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.151684    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"]
Sep 16 01:00:49.155082 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.155063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:49.179438 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.179412    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"]
Sep 16 01:00:49.201097 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.201067    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8400e400-f206-4708-a85c-d64d5e8da8a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g\" (UID: \"8400e400-f206-4708-a85c-d64d5e8da8a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:49.201217 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.201108    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8w2\" (UniqueName: \"kubernetes.io/projected/8400e400-f206-4708-a85c-d64d5e8da8a0-kube-api-access-8x8w2\") pod \"payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g\" (UID: \"8400e400-f206-4708-a85c-d64d5e8da8a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:49.302567 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.302507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8400e400-f206-4708-a85c-d64d5e8da8a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g\" (UID: \"8400e400-f206-4708-a85c-d64d5e8da8a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:49.303002 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.302580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8w2\" (UniqueName: \"kubernetes.io/projected/8400e400-f206-4708-a85c-d64d5e8da8a0-kube-api-access-8x8w2\") pod \"payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g\" (UID: \"8400e400-f206-4708-a85c-d64d5e8da8a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:49.303002 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.302637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75zp\" (UniqueName: \"kubernetes.io/projected/64424d37-f681-416b-ac15-0616e5c064df-kube-api-access-x75zp\") pod \"payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6\" (UID: \"64424d37-f681-416b-ac15-0616e5c064df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:49.303002 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.302658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64424d37-f681-416b-ac15-0616e5c064df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6\" (UID: \"64424d37-f681-416b-ac15-0616e5c064df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:49.303475 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.303451    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8400e400-f206-4708-a85c-d64d5e8da8a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g\" (UID: \"8400e400-f206-4708-a85c-d64d5e8da8a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:49.310145 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.310126    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8w2\" (UniqueName: \"kubernetes.io/projected/8400e400-f206-4708-a85c-d64d5e8da8a0-kube-api-access-8x8w2\") pod \"payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g\" (UID: \"8400e400-f206-4708-a85c-d64d5e8da8a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:49.361644 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.361608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47" event={"ID":"26927ad2-6d42-4ab7-bde1-6d363b14afcd","Type":"ContainerStarted","Data":"a85c6542551d9bd852d010bc78644103c54d677e75ac03d43738c989aff05362"}
Sep 16 01:00:49.361823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.361661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47" event={"ID":"26927ad2-6d42-4ab7-bde1-6d363b14afcd","Type":"ContainerStarted","Data":"cea6bd38e9e13001a6c48fb1de5de9530ffddd1ad11613ec80128028f6bd97cb"}
Sep 16 01:00:49.361823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.361740    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:00:49.381468 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.381379    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47" podStartSLOduration=1.381361446 podStartE2EDuration="1.381361446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:49.379189798 +0000 UTC m=+1035.082011994" watchObservedRunningTime="2026-09-16 01:00:49.381361446 +0000 UTC m=+1035.084183640"
Sep 16 01:00:49.403446 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.403420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75zp\" (UniqueName: \"kubernetes.io/projected/64424d37-f681-416b-ac15-0616e5c064df-kube-api-access-x75zp\") pod \"payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6\" (UID: \"64424d37-f681-416b-ac15-0616e5c064df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:49.403576 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.403462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64424d37-f681-416b-ac15-0616e5c064df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6\" (UID: \"64424d37-f681-416b-ac15-0616e5c064df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:49.404321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.404302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64424d37-f681-416b-ac15-0616e5c064df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6\" (UID: \"64424d37-f681-416b-ac15-0616e5c064df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:49.411288 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.411266    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75zp\" (UniqueName: \"kubernetes.io/projected/64424d37-f681-416b-ac15-0616e5c064df-kube-api-access-x75zp\") pod \"payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6\" (UID: \"64424d37-f681-416b-ac15-0616e5c064df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:49.441199 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.441176    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:49.476265 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.476236    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:49.725387 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.725362    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"]
Sep 16 01:00:49.729105 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:00:49.728871    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8400e400_f206_4708_a85c_d64d5e8da8a0.slice/crio-aaf54b4b4f4240129ddaeb648b8eb9db96c19277fc7e2246c2e56b7706da8679 WatchSource:0}: Error finding container aaf54b4b4f4240129ddaeb648b8eb9db96c19277fc7e2246c2e56b7706da8679: Status 404 returned error can't find the container with id aaf54b4b4f4240129ddaeb648b8eb9db96c19277fc7e2246c2e56b7706da8679
Sep 16 01:00:49.769691 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:49.769669    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"]
Sep 16 01:00:49.773738 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:00:49.773711    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64424d37_f681_416b_ac15_0616e5c064df.slice/crio-6ea5eafccf5a21ff82f55f7a319dd429ee13c4bd7b5ca42ac20ac68a13c5efe9 WatchSource:0}: Error finding container 6ea5eafccf5a21ff82f55f7a319dd429ee13c4bd7b5ca42ac20ac68a13c5efe9: Status 404 returned error can't find the container with id 6ea5eafccf5a21ff82f55f7a319dd429ee13c4bd7b5ca42ac20ac68a13c5efe9
Sep 16 01:00:50.373639 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:50.373564    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g" event={"ID":"8400e400-f206-4708-a85c-d64d5e8da8a0","Type":"ContainerStarted","Data":"2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8"}
Sep 16 01:00:50.374288 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:50.374266    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:00:50.374471 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:50.374453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g" event={"ID":"8400e400-f206-4708-a85c-d64d5e8da8a0","Type":"ContainerStarted","Data":"aaf54b4b4f4240129ddaeb648b8eb9db96c19277fc7e2246c2e56b7706da8679"}
Sep 16 01:00:50.375668 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:50.375642    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6" event={"ID":"64424d37-f681-416b-ac15-0616e5c064df","Type":"ContainerStarted","Data":"d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982"}
Sep 16 01:00:50.375797 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:50.375688    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6" event={"ID":"64424d37-f681-416b-ac15-0616e5c064df","Type":"ContainerStarted","Data":"6ea5eafccf5a21ff82f55f7a319dd429ee13c4bd7b5ca42ac20ac68a13c5efe9"}
Sep 16 01:00:50.375797 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:50.375792    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:00:50.388455 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:50.388407    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g" podStartSLOduration=1.388388938 podStartE2EDuration="1.388388938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:50.388100693 +0000 UTC m=+1036.090922890" watchObservedRunningTime="2026-09-16 01:00:50.388388938 +0000 UTC m=+1036.091211133"
Sep 16 01:00:50.405264 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:00:50.405211    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6" podStartSLOduration=1.40519365 podStartE2EDuration="1.40519365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:50.402559194 +0000 UTC m=+1036.105381390" watchObservedRunningTime="2026-09-16 01:00:50.40519365 +0000 UTC m=+1036.108015848"
Sep 16 01:01:00.369766 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:00.369718    2573 patch_prober.go:28] interesting pod/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 01:01:00.370252 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:00.369792    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 01:01:00.370252 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:00.369715    2573 patch_prober.go:28] interesting pod/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 01:01:00.370252 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:00.369873    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 01:01:00.370708 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:00.370681    2573 patch_prober.go:28] interesting pod/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 01:01:00.370858 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:00.370726    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 01:01:00.378506 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:00.378469    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:01:01.385491 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:01.385454    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:01:02.390275 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:02.390243    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:01:04.470576 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.470490    2573 generic.go:356] "Generic (PLEG): container finished" podID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerID="6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504" exitCode=0
Sep 16 01:01:04.471008 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.470571    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" event={"ID":"87589a4b-1114-4d80-91b3-34f5270b9b6d","Type":"ContainerDied","Data":"6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504"}
Sep 16 01:01:04.471081 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.471063    2573 scope.go:117] "RemoveContainer" containerID="6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504"
Sep 16 01:01:04.472279 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.472247    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:04.472362 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.472319    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:04.472362 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.472357    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:04.472439 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.472360    2573 generic.go:356] "Generic (PLEG): container finished" podID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerID="58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2" exitCode=0
Sep 16 01:01:04.472481 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.472440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" event={"ID":"c67c81bd-f337-446c-9bce-977dd0fb592a","Type":"ContainerDied","Data":"58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2"}
Sep 16 01:01:04.472886 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.472872    2573 scope.go:117] "RemoveContainer" containerID="58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2"
Sep 16 01:01:04.474085 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.474059    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:04.474178 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.474135    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:04.474237 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.474178    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:04.474402 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.474386    2573 generic.go:356] "Generic (PLEG): container finished" podID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerID="26554cfce7e1999be01a3336217853dd9578d2c4b2bd178618ad14513a11075d" exitCode=0
Sep 16 01:01:04.474494 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.474471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" event={"ID":"06460b71-72f9-4fd3-93f8-d61f6ecd5be0","Type":"ContainerDied","Data":"26554cfce7e1999be01a3336217853dd9578d2c4b2bd178618ad14513a11075d"}
Sep 16 01:01:04.474820 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.474805    2573 scope.go:117] "RemoveContainer" containerID="26554cfce7e1999be01a3336217853dd9578d2c4b2bd178618ad14513a11075d"
Sep 16 01:01:04.475880 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.475850    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:04.475969 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.475927    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:04.476014 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:04.475968    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:05.485701 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:05.485650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" event={"ID":"87589a4b-1114-4d80-91b3-34f5270b9b6d","Type":"ContainerStarted","Data":"c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d"}
Sep 16 01:01:05.488948 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:05.488913    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" event={"ID":"c67c81bd-f337-446c-9bce-977dd0fb592a","Type":"ContainerStarted","Data":"10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935"}
Sep 16 01:01:05.490895 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:05.490872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" event={"ID":"06460b71-72f9-4fd3-93f8-d61f6ecd5be0","Type":"ContainerStarted","Data":"d7e425093b7f57ac7d0a26b509cbcb500e08e6fbf91f5baefce54fab02711e02"}
Sep 16 01:01:05.513430 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:05.513400    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:01:05.513619 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:05.513598    2573 patch_prober.go:28] interesting pod/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:01:05.513706 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:05.513653    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 01:01:06.398345 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:06.398304    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:01:06.398578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:06.398438    2573 patch_prober.go:28] interesting pod/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 01:01:06.398578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:06.398484    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 01:01:06.447876 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:06.447827    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:01:06.448320 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:06.448291    2573 patch_prober.go:28] interesting pod/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 01:01:06.448537 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:06.448490    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 01:01:06.513408 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:06.513381    2573 patch_prober.go:28] interesting pod/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:01:06.513881 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:06.513590    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 01:01:07.398240 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:07.398204    2573 patch_prober.go:28] interesting pod/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 01:01:07.398535 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:07.398487    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 01:01:07.448283 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:07.448247    2573 patch_prober.go:28] interesting pod/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 01:01:07.448536 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:07.448489    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 01:01:07.513656 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:07.513628    2573 patch_prober.go:28] interesting pod/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:01:07.514133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:07.513733    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 01:01:08.398218 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:08.398172    2573 patch_prober.go:28] interesting pod/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 01:01:08.398431 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:08.398243    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 01:01:08.448366 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:08.448329    2573 patch_prober.go:28] interesting pod/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 01:01:08.448686 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:08.448643    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 01:01:08.512862 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:08.512831    2573 patch_prober.go:28] interesting pod/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:01:08.513051 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:08.512882    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 01:01:09.398546 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:09.398485    2573 patch_prober.go:28] interesting pod/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 01:01:09.399171 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:09.399131    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 01:01:09.447977 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:09.447927    2573 patch_prober.go:28] interesting pod/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 01:01:09.448275 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:09.448233    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 01:01:09.513727 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:09.513693    2573 patch_prober.go:28] interesting pod/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:01:09.513909 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:09.513779    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 01:01:10.398316 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:10.398268    2573 patch_prober.go:28] interesting pod/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 16 01:01:10.398564 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:10.398342    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 16 01:01:10.447893 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:10.447848    2573 patch_prober.go:28] interesting pod/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 01:01:10.448094 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:10.447932    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 01:01:10.513759 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:10.513720    2573 patch_prober.go:28] interesting pod/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 01:01:10.513945 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:10.513775    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 01:01:11.452617 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.452586    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:01:11.453057 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.452872    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:01:11.453586 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.453570    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:01:11.519411 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.519332    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:01:11.519796 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.519770    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:01:11.520927 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.520907    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:01:11.677632 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.677589    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:01:11.678045 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.678024    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:01:11.681696 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:11.681626    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:01:23.765198 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:23.765165    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:01:33.765509 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:33.765429    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:01:38.281939 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.281908    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"]
Sep 16 01:01:38.284740 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.284722    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.287898 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.287874    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"simulator-w3-2534e3-kserve-self-signed-certs\""
Sep 16 01:01:38.288038 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.287898    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"kube-root-ca.crt\""
Sep 16 01:01:38.288038 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.287898    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"openshift-service-ca.crt\""
Sep 16 01:01:38.288038 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.287876    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"default-dockercfg-t7hw2\""
Sep 16 01:01:38.295192 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.295173    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"]
Sep 16 01:01:38.339776 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.339752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-model-cache\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.339904 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.339795    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-home\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.339904 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.339836    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-dshm\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.339904 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.339875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00dd3601-90c7-4ac2-94a5-115d870b78ee-tls-certs\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.339904 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.339899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-tmp-dir\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.340064 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.339918    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcpr\" (UniqueName: \"kubernetes.io/projected/00dd3601-90c7-4ac2-94a5-115d870b78ee-kube-api-access-7fcpr\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441230 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-model-cache\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441386 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-home\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441386 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-dshm\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441514 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00dd3601-90c7-4ac2-94a5-115d870b78ee-tls-certs\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441514 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-tmp-dir\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441673 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441591    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-home\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441673 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcpr\" (UniqueName: \"kubernetes.io/projected/00dd3601-90c7-4ac2-94a5-115d870b78ee-kube-api-access-7fcpr\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441665    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-model-cache\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.441842 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.441824    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-tmp-dir\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.443907 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.443882    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-dshm\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.444031 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.443984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00dd3601-90c7-4ac2-94a5-115d870b78ee-tls-certs\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.451560 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.450465    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcpr\" (UniqueName: \"kubernetes.io/projected/00dd3601-90c7-4ac2-94a5-115d870b78ee-kube-api-access-7fcpr\") pod \"simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") " pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.594781 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.594706    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:38.870791 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:38.870766    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"]
Sep 16 01:01:38.872656 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:01:38.872627    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00dd3601_90c7_4ac2_94a5_115d870b78ee.slice/crio-07e659105bcec2335af285d38e9b1423047d785136f5e6a350bf57d076647abf WatchSource:0}: Error finding container 07e659105bcec2335af285d38e9b1423047d785136f5e6a350bf57d076647abf: Status 404 returned error can't find the container with id 07e659105bcec2335af285d38e9b1423047d785136f5e6a350bf57d076647abf
Sep 16 01:01:39.693085 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:39.693044    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh" event={"ID":"00dd3601-90c7-4ac2-94a5-115d870b78ee","Type":"ContainerStarted","Data":"d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5"}
Sep 16 01:01:39.693453 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:39.693094    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh" event={"ID":"00dd3601-90c7-4ac2-94a5-115d870b78ee","Type":"ContainerStarted","Data":"07e659105bcec2335af285d38e9b1423047d785136f5e6a350bf57d076647abf"}
Sep 16 01:01:39.710351 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:39.710309    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh" podStartSLOduration=1.710294449 podStartE2EDuration="1.710294449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:39.708062944 +0000 UTC m=+1085.410885138" watchObservedRunningTime="2026-09-16 01:01:39.710294449 +0000 UTC m=+1085.413116645"
Sep 16 01:01:48.595099 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:48.595064    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:48.595099 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:48.595106    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:48.607228 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:48.607206    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:01:48.755214 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:01:48.755182    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:02:43.765825 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:02:43.765784    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:02:45.764821 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:02:45.764788    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:03:23.686150 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.686113    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"]
Sep 16 01:03:23.689057 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.689042    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.692233 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.692206    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6d0d72\"/\"openshift-service-ca.crt\""
Sep 16 01:03:23.692375 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.692249    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6d0d72\"/\"default-dockercfg-9p9w6\""
Sep 16 01:03:23.692375 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.692279    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6d0d72\"/\"test-model-6d0d72-kserve-self-signed-certs\""
Sep 16 01:03:23.692565 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.692550    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6d0d72\"/\"kube-root-ca.crt\""
Sep 16 01:03:23.698315 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.697474    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"]
Sep 16 01:03:23.764004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.763978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-model-cache\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.764004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.764013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de64c3e-9138-4b4c-a004-8d314a6c3778-tls-certs\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.764193 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.764033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-tmp-dir\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.764193 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.764141    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-home\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.764193 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.764184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-dshm\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.764335 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.764209    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzfr7\" (UniqueName: \"kubernetes.io/projected/8de64c3e-9138-4b4c-a004-8d314a6c3778-kube-api-access-jzfr7\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.864965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.864935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-model-cache\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.865121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.864974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de64c3e-9138-4b4c-a004-8d314a6c3778-tls-certs\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.865121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.865001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-tmp-dir\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.865121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.865046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-home\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.865121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.865079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-dshm\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.865121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.865112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzfr7\" (UniqueName: \"kubernetes.io/projected/8de64c3e-9138-4b4c-a004-8d314a6c3778-kube-api-access-jzfr7\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.865439 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.865408    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-model-cache\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.865492 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.865457    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-tmp-dir\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.865670 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.865647    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-home\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.867546 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.867510    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-dshm\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.867741 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.867726    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de64c3e-9138-4b4c-a004-8d314a6c3778-tls-certs\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:23.872674 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:23.872653    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzfr7\" (UniqueName: \"kubernetes.io/projected/8de64c3e-9138-4b4c-a004-8d314a6c3778-kube-api-access-jzfr7\") pod \"test-model-6d0d72-kserve-77d846d67f-mbg5l\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") " pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:24.001131 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:24.001096    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:24.278172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:24.278149    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"]
Sep 16 01:03:24.280051 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:03:24.280024    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8de64c3e_9138_4b4c_a004_8d314a6c3778.slice/crio-d4007c18daf29cc6e71b3b0f60d6b601b0ceb2246a11d3c96b898cb24b6ddd72 WatchSource:0}: Error finding container d4007c18daf29cc6e71b3b0f60d6b601b0ceb2246a11d3c96b898cb24b6ddd72: Status 404 returned error can't find the container with id d4007c18daf29cc6e71b3b0f60d6b601b0ceb2246a11d3c96b898cb24b6ddd72
Sep 16 01:03:25.278565 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:25.278510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l" event={"ID":"8de64c3e-9138-4b4c-a004-8d314a6c3778","Type":"ContainerStarted","Data":"c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db"}
Sep 16 01:03:25.278565 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:25.278569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l" event={"ID":"8de64c3e-9138-4b4c-a004-8d314a6c3778","Type":"ContainerStarted","Data":"d4007c18daf29cc6e71b3b0f60d6b601b0ceb2246a11d3c96b898cb24b6ddd72"}
Sep 16 01:03:25.294657 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:25.294609    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l" podStartSLOduration=2.2945939539999998 podStartE2EDuration="2.294593954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:03:25.293446362 +0000 UTC m=+1190.996268557" watchObservedRunningTime="2026-09-16 01:03:25.294593954 +0000 UTC m=+1190.997416148"
Sep 16 01:03:34.001501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:34.001465    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:34.001501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:34.001505    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:34.014439 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:34.014417    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:34.342909 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:34.342826    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:03:50.132493 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.132462    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"]
Sep 16 01:03:50.135202 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.135185    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.138742 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.138721    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2534e3\"/\"distinct-w3-2534e3-kserve-self-signed-certs\""
Sep 16 01:03:50.163612 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.163591    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"]
Sep 16 01:03:50.225902 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.225875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngf2\" (UniqueName: \"kubernetes.io/projected/2e74ae40-4e63-49f0-a734-d41735891c52-kube-api-access-hngf2\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.226052 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.225991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-tmp-dir\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.226098 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.226059    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-home\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.226098 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.226075    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e74ae40-4e63-49f0-a734-d41735891c52-tls-certs\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.226098 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.226091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-model-cache\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.226214 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.226119    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-dshm\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.326766 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.326730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e74ae40-4e63-49f0-a734-d41735891c52-tls-certs\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.326766 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.326767    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-model-cache\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.326983 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.326802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-dshm\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.326983 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.326869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngf2\" (UniqueName: \"kubernetes.io/projected/2e74ae40-4e63-49f0-a734-d41735891c52-kube-api-access-hngf2\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.326983 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.326967    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-tmp-dir\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.327136 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.327060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-home\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.327200 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.327186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-model-cache\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.327335 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.327312    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-tmp-dir\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.327409 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.327368    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-home\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.333582 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.330196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e74ae40-4e63-49f0-a734-d41735891c52-tls-certs\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.337539 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.334385    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-dshm\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.338134 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.338112    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngf2\" (UniqueName: \"kubernetes.io/projected/2e74ae40-4e63-49f0-a734-d41735891c52-kube-api-access-hngf2\") pod \"distinct-w3-2534e3-kserve-6944dd8998-64b46\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") " pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.443933 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.443865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:03:50.717481 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:50.717456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"]
Sep 16 01:03:50.719332 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:03:50.719308    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e74ae40_4e63_49f0_a734_d41735891c52.slice/crio-9445b9e9cbd8ab1823e37bfa35650b558111075cf7c7a7da0eaa998af20bc542 WatchSource:0}: Error finding container 9445b9e9cbd8ab1823e37bfa35650b558111075cf7c7a7da0eaa998af20bc542: Status 404 returned error can't find the container with id 9445b9e9cbd8ab1823e37bfa35650b558111075cf7c7a7da0eaa998af20bc542
Sep 16 01:03:51.426935 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:51.426900    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46" event={"ID":"2e74ae40-4e63-49f0-a734-d41735891c52","Type":"ContainerStarted","Data":"eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5"}
Sep 16 01:03:51.426935 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:51.426946    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46" event={"ID":"2e74ae40-4e63-49f0-a734-d41735891c52","Type":"ContainerStarted","Data":"9445b9e9cbd8ab1823e37bfa35650b558111075cf7c7a7da0eaa998af20bc542"}
Sep 16 01:03:51.442893 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:51.442837    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46" podStartSLOduration=1.442819613 podStartE2EDuration="1.442819613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:03:51.441840413 +0000 UTC m=+1217.144662609" watchObservedRunningTime="2026-09-16 01:03:51.442819613 +0000 UTC m=+1217.145641810"
Sep 16 01:03:55.765076 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:03:55.765044    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:04:00.444598 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:00.444567    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:04:00.445299 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:00.444613    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:04:00.457471 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:00.457449    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:04:00.489934 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:00.489906    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:04:07.765740 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:07.765708    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:04:54.891709 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:54.891620    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"]
Sep 16 01:04:54.895351 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:54.895336    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:54.897635 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:54.897609    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f097ba39\""
Sep 16 01:04:54.897761 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:54.897657    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f097ba39-dockercfg-chshg\""
Sep 16 01:04:54.905142 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:54.905122    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"]
Sep 16 01:04:54.968859 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:54.968826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d734a671-4deb-4030-854c-c27d803f8e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz\" (UID: \"d734a671-4deb-4030-854c-c27d803f8e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:54.969031 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:54.968941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bm4n\" (UniqueName: \"kubernetes.io/projected/d734a671-4deb-4030-854c-c27d803f8e5b-kube-api-access-5bm4n\") pod \"payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz\" (UID: \"d734a671-4deb-4030-854c-c27d803f8e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:55.070078 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.070047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bm4n\" (UniqueName: \"kubernetes.io/projected/d734a671-4deb-4030-854c-c27d803f8e5b-kube-api-access-5bm4n\") pod \"payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz\" (UID: \"d734a671-4deb-4030-854c-c27d803f8e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:55.070271 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.070158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d734a671-4deb-4030-854c-c27d803f8e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz\" (UID: \"d734a671-4deb-4030-854c-c27d803f8e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:55.070874 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.070849    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d734a671-4deb-4030-854c-c27d803f8e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz\" (UID: \"d734a671-4deb-4030-854c-c27d803f8e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:55.077925 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.077902    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bm4n\" (UniqueName: \"kubernetes.io/projected/d734a671-4deb-4030-854c-c27d803f8e5b-kube-api-access-5bm4n\") pod \"payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz\" (UID: \"d734a671-4deb-4030-854c-c27d803f8e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:55.207115 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.207074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:55.485136 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.485112    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"]
Sep 16 01:04:55.486464 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:04:55.486438    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd734a671_4deb_4030_854c_c27d803f8e5b.slice/crio-be88da0577e065cd0839f56ba2f924bf8e5abe1d189ffbf31e7841ad992f86f5 WatchSource:0}: Error finding container be88da0577e065cd0839f56ba2f924bf8e5abe1d189ffbf31e7841ad992f86f5: Status 404 returned error can't find the container with id be88da0577e065cd0839f56ba2f924bf8e5abe1d189ffbf31e7841ad992f86f5
Sep 16 01:04:55.792804 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.792714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz" event={"ID":"d734a671-4deb-4030-854c-c27d803f8e5b","Type":"ContainerStarted","Data":"51b672f5dd423751f8412b42972cbfbda3f7734dbc11646e91c547d39d670d13"}
Sep 16 01:04:55.792804 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.792767    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:04:55.792804 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.792784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz" event={"ID":"d734a671-4deb-4030-854c-c27d803f8e5b","Type":"ContainerStarted","Data":"be88da0577e065cd0839f56ba2f924bf8e5abe1d189ffbf31e7841ad992f86f5"}
Sep 16 01:04:55.809042 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:04:55.808990    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz" podStartSLOduration=1.808975544 podStartE2EDuration="1.808975544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:04:55.807314682 +0000 UTC m=+1281.510136878" watchObservedRunningTime="2026-09-16 01:04:55.808975544 +0000 UTC m=+1281.511797738"
Sep 16 01:05:00.765706 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:00.765674    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:05:06.800166 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:06.800138    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:05:22.193474 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.193441    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"]
Sep 16 01:05:22.197174 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.197158    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"
Sep 16 01:05:22.199317 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.199297    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:05:22.199422 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.199299    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:05:22.203747 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.203724    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"]
Sep 16 01:05:22.239952 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.239921    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc126d-8902-4ce6-97b0-b4c47d8a645b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4\" (UID: \"8efc126d-8902-4ce6-97b0-b4c47d8a645b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"
Sep 16 01:05:22.341146 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.341114    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc126d-8902-4ce6-97b0-b4c47d8a645b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4\" (UID: \"8efc126d-8902-4ce6-97b0-b4c47d8a645b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"
Sep 16 01:05:22.341742 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.341722    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc126d-8902-4ce6-97b0-b4c47d8a645b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4\" (UID: \"8efc126d-8902-4ce6-97b0-b4c47d8a645b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"
Sep 16 01:05:22.510589 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.510563    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"
Sep 16 01:05:22.785095 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.785071    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"]
Sep 16 01:05:22.786538 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:05:22.786493    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efc126d_8902_4ce6_97b0_b4c47d8a645b.slice/crio-5ea4966de26b848e6df16d1dfb0c5cf55c9bcb0ace1a4ef3791ff829d5dba645 WatchSource:0}: Error finding container 5ea4966de26b848e6df16d1dfb0c5cf55c9bcb0ace1a4ef3791ff829d5dba645: Status 404 returned error can't find the container with id 5ea4966de26b848e6df16d1dfb0c5cf55c9bcb0ace1a4ef3791ff829d5dba645
Sep 16 01:05:22.941878 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:22.941848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4" event={"ID":"8efc126d-8902-4ce6-97b0-b4c47d8a645b","Type":"ContainerStarted","Data":"5ea4966de26b848e6df16d1dfb0c5cf55c9bcb0ace1a4ef3791ff829d5dba645"}
Sep 16 01:05:23.949319 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:23.949287    2573 generic.go:356] "Generic (PLEG): container finished" podID="8efc126d-8902-4ce6-97b0-b4c47d8a645b" containerID="d77dd8acc75e3803a7e7b9ba79c7c290ac21d57350bd2486688054a455a0bf4f" exitCode=0
Sep 16 01:05:23.949718 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:23.949373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4" event={"ID":"8efc126d-8902-4ce6-97b0-b4c47d8a645b","Type":"ContainerDied","Data":"d77dd8acc75e3803a7e7b9ba79c7c290ac21d57350bd2486688054a455a0bf4f"}
Sep 16 01:05:24.987547 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:24.987495    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"]
Sep 16 01:05:24.991627 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:24.991605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"
Sep 16 01:05:24.997560 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:24.997504    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"]
Sep 16 01:05:25.068430 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.068400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ca788-897c-4cb6-8499-adcd63b2953e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d58169-49bqk\" (UID: \"594ca788-897c-4cb6-8499-adcd63b2953e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"
Sep 16 01:05:25.118355 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.118335    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"
Sep 16 01:05:25.169805 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.169780    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc126d-8902-4ce6-97b0-b4c47d8a645b-maas-api-service-ca\") pod \"8efc126d-8902-4ce6-97b0-b4c47d8a645b\" (UID: \"8efc126d-8902-4ce6-97b0-b4c47d8a645b\") "
Sep 16 01:05:25.170028 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.170012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ca788-897c-4cb6-8499-adcd63b2953e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d58169-49bqk\" (UID: \"594ca788-897c-4cb6-8499-adcd63b2953e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"
Sep 16 01:05:25.170219 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.170192    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efc126d-8902-4ce6-97b0-b4c47d8a645b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8efc126d-8902-4ce6-97b0-b4c47d8a645b" (UID: "8efc126d-8902-4ce6-97b0-b4c47d8a645b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:25.170847 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.170825    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ca788-897c-4cb6-8499-adcd63b2953e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d58169-49bqk\" (UID: \"594ca788-897c-4cb6-8499-adcd63b2953e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"
Sep 16 01:05:25.271321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.271260    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc126d-8902-4ce6-97b0-b4c47d8a645b-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:25.305759 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.305739    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"
Sep 16 01:05:25.577447 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.577427    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"]
Sep 16 01:05:25.579831 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:05:25.579805    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594ca788_897c_4cb6_8499_adcd63b2953e.slice/crio-7a15872312b84ffad4b3efd55d9fb892713b1feff6fd9d3f215b2f1d7648119b WatchSource:0}: Error finding container 7a15872312b84ffad4b3efd55d9fb892713b1feff6fd9d3f215b2f1d7648119b: Status 404 returned error can't find the container with id 7a15872312b84ffad4b3efd55d9fb892713b1feff6fd9d3f215b2f1d7648119b
Sep 16 01:05:25.962030 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.962005    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"
Sep 16 01:05:25.962219 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.962031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4" event={"ID":"8efc126d-8902-4ce6-97b0-b4c47d8a645b","Type":"ContainerDied","Data":"5ea4966de26b848e6df16d1dfb0c5cf55c9bcb0ace1a4ef3791ff829d5dba645"}
Sep 16 01:05:25.962219 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.962081    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea4966de26b848e6df16d1dfb0c5cf55c9bcb0ace1a4ef3791ff829d5dba645"
Sep 16 01:05:25.963760 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.963736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk" event={"ID":"594ca788-897c-4cb6-8499-adcd63b2953e","Type":"ContainerStarted","Data":"67c8ae656d6b59dd9c1dedbf6248de23d54482e2c9fdb5b4e80a544bc86da665"}
Sep 16 01:05:25.963868 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.963786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk" event={"ID":"594ca788-897c-4cb6-8499-adcd63b2953e","Type":"ContainerStarted","Data":"7a15872312b84ffad4b3efd55d9fb892713b1feff6fd9d3f215b2f1d7648119b"}
Sep 16 01:05:25.980719 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:25.980682    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk" podStartSLOduration=1.98066946 podStartE2EDuration="1.98066946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:05:25.978320866 +0000 UTC m=+1311.681143064" watchObservedRunningTime="2026-09-16 01:05:25.98066946 +0000 UTC m=+1311.683491655"
Sep 16 01:05:26.970298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:26.970258    2573 generic.go:356] "Generic (PLEG): container finished" podID="594ca788-897c-4cb6-8499-adcd63b2953e" containerID="67c8ae656d6b59dd9c1dedbf6248de23d54482e2c9fdb5b4e80a544bc86da665" exitCode=0
Sep 16 01:05:26.970828 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:26.970334    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk" event={"ID":"594ca788-897c-4cb6-8499-adcd63b2953e","Type":"ContainerDied","Data":"67c8ae656d6b59dd9c1dedbf6248de23d54482e2c9fdb5b4e80a544bc86da665"}
Sep 16 01:05:28.152252 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:28.152230    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"
Sep 16 01:05:28.199040 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:28.199004    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ca788-897c-4cb6-8499-adcd63b2953e-maas-api-service-ca\") pod \"594ca788-897c-4cb6-8499-adcd63b2953e\" (UID: \"594ca788-897c-4cb6-8499-adcd63b2953e\") "
Sep 16 01:05:28.199330 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:28.199306    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594ca788-897c-4cb6-8499-adcd63b2953e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "594ca788-897c-4cb6-8499-adcd63b2953e" (UID: "594ca788-897c-4cb6-8499-adcd63b2953e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:28.300754 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:28.300681    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/594ca788-897c-4cb6-8499-adcd63b2953e-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:28.765375 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:28.765343    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:05:28.983069 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:28.983038    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"
Sep 16 01:05:28.983264 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:28.983083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk" event={"ID":"594ca788-897c-4cb6-8499-adcd63b2953e","Type":"ContainerDied","Data":"7a15872312b84ffad4b3efd55d9fb892713b1feff6fd9d3f215b2f1d7648119b"}
Sep 16 01:05:28.983264 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:28.983155    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a15872312b84ffad4b3efd55d9fb892713b1feff6fd9d3f215b2f1d7648119b"
Sep 16 01:05:30.786562 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.786505    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"]
Sep 16 01:05:30.786982 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.786969    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efc126d-8902-4ce6-97b0-b4c47d8a645b" containerName="revoke-keys"
Sep 16 01:05:30.787023 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.786984    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efc126d-8902-4ce6-97b0-b4c47d8a645b" containerName="revoke-keys"
Sep 16 01:05:30.787023 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.786994    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594ca788-897c-4cb6-8499-adcd63b2953e" containerName="revoke-keys"
Sep 16 01:05:30.787023 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.786999    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="594ca788-897c-4cb6-8499-adcd63b2953e" containerName="revoke-keys"
Sep 16 01:05:30.787121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.787076    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="594ca788-897c-4cb6-8499-adcd63b2953e" containerName="revoke-keys"
Sep 16 01:05:30.787121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.787087    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8efc126d-8902-4ce6-97b0-b4c47d8a645b" containerName="revoke-keys"
Sep 16 01:05:30.790363 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.790348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:30.792539 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.792507    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"openshift-service-ca.crt\""
Sep 16 01:05:30.793289 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.793268    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"kube-root-ca.crt\""
Sep 16 01:05:30.793386 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.793317    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"default-dockercfg-67hvz\""
Sep 16 01:05:30.793386 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.793354    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"distinct-2-w0-96c840-kserve-self-signed-certs\""
Sep 16 01:05:30.800066 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.800046    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"]
Sep 16 01:05:30.924868 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.924835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-model-cache\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:30.925069 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.924879    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4976c5c6-a606-43ec-b334-edd7b718a053-tls-certs\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:30.925069 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.924977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xhc\" (UniqueName: \"kubernetes.io/projected/4976c5c6-a606-43ec-b334-edd7b718a053-kube-api-access-d9xhc\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:30.925069 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.925024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-home\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:30.925178 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.925067    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-dshm\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:30.925178 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:30.925109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-tmp-dir\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.025993 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.025948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xhc\" (UniqueName: \"kubernetes.io/projected/4976c5c6-a606-43ec-b334-edd7b718a053-kube-api-access-d9xhc\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.026215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.026018    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-home\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.026215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.026065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-dshm\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.026215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.026102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-tmp-dir\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.026215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.026157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-model-cache\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.026215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.026188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4976c5c6-a606-43ec-b334-edd7b718a053-tls-certs\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.026473 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.026440    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-home\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.026561 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.026475    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-model-cache\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.026609 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.026585    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-tmp-dir\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.028514 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.028483    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-dshm\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.029055 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.029035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4976c5c6-a606-43ec-b334-edd7b718a053-tls-certs\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.033850 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.033822    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xhc\" (UniqueName: \"kubernetes.io/projected/4976c5c6-a606-43ec-b334-edd7b718a053-kube-api-access-d9xhc\") pod \"distinct-2-w0-96c840-kserve-56754cffc6-99t9q\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") " pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.099871 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.099777    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:31.377355 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:31.377329    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"]
Sep 16 01:05:31.378939 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:05:31.378908    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4976c5c6_a606_43ec_b334_edd7b718a053.slice/crio-f21e2beb31ed63486ea085fcbf58a1be2eecd8a83e7f387f40ba6dd02eb82e56 WatchSource:0}: Error finding container f21e2beb31ed63486ea085fcbf58a1be2eecd8a83e7f387f40ba6dd02eb82e56: Status 404 returned error can't find the container with id f21e2beb31ed63486ea085fcbf58a1be2eecd8a83e7f387f40ba6dd02eb82e56
Sep 16 01:05:32.002618 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:32.002579    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q" event={"ID":"4976c5c6-a606-43ec-b334-edd7b718a053","Type":"ContainerStarted","Data":"2525eba794e15b2f491d36d69ece18febccb3c432371a7cf274b1c93dc2ef5cb"}
Sep 16 01:05:32.003002 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:32.002628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q" event={"ID":"4976c5c6-a606-43ec-b334-edd7b718a053","Type":"ContainerStarted","Data":"f21e2beb31ed63486ea085fcbf58a1be2eecd8a83e7f387f40ba6dd02eb82e56"}
Sep 16 01:05:32.019790 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:32.019724    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q" podStartSLOduration=2.019708304 podStartE2EDuration="2.019708304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:05:32.016985148 +0000 UTC m=+1317.719807356" watchObservedRunningTime="2026-09-16 01:05:32.019708304 +0000 UTC m=+1317.722530528"
Sep 16 01:05:33.179119 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:33.179086    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"]
Sep 16 01:05:33.770195 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:33.770159    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"]
Sep 16 01:05:33.770557 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:33.770485    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz" podUID="d734a671-4deb-4030-854c-c27d803f8e5b" containerName="payload-pre-processing" containerID="cri-o://51b672f5dd423751f8412b42972cbfbda3f7734dbc11646e91c547d39d670d13" gracePeriod=30
Sep 16 01:05:33.971734 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:33.971671    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"]
Sep 16 01:05:33.975275 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:33.975244    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f097ba39-rcwj4"]
Sep 16 01:05:34.015656 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.015622    2573 generic.go:356] "Generic (PLEG): container finished" podID="d734a671-4deb-4030-854c-c27d803f8e5b" containerID="51b672f5dd423751f8412b42972cbfbda3f7734dbc11646e91c547d39d670d13" exitCode=0
Sep 16 01:05:34.015777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.015686    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz" event={"ID":"d734a671-4deb-4030-854c-c27d803f8e5b","Type":"ContainerDied","Data":"51b672f5dd423751f8412b42972cbfbda3f7734dbc11646e91c547d39d670d13"}
Sep 16 01:05:34.048058 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.048029    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:05:34.162646 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.162611    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d734a671-4deb-4030-854c-c27d803f8e5b-plugins-config-volume\") pod \"d734a671-4deb-4030-854c-c27d803f8e5b\" (UID: \"d734a671-4deb-4030-854c-c27d803f8e5b\") "
Sep 16 01:05:34.162816 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.162756    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bm4n\" (UniqueName: \"kubernetes.io/projected/d734a671-4deb-4030-854c-c27d803f8e5b-kube-api-access-5bm4n\") pod \"d734a671-4deb-4030-854c-c27d803f8e5b\" (UID: \"d734a671-4deb-4030-854c-c27d803f8e5b\") "
Sep 16 01:05:34.162968 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.162945    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d734a671-4deb-4030-854c-c27d803f8e5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d734a671-4deb-4030-854c-c27d803f8e5b" (UID: "d734a671-4deb-4030-854c-c27d803f8e5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:34.165074 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.165050    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d734a671-4deb-4030-854c-c27d803f8e5b-kube-api-access-5bm4n" (OuterVolumeSpecName: "kube-api-access-5bm4n") pod "d734a671-4deb-4030-854c-c27d803f8e5b" (UID: "d734a671-4deb-4030-854c-c27d803f8e5b"). InnerVolumeSpecName "kube-api-access-5bm4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:34.263744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.263709    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bm4n\" (UniqueName: \"kubernetes.io/projected/d734a671-4deb-4030-854c-c27d803f8e5b-kube-api-access-5bm4n\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:34.263744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.263735    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d734a671-4deb-4030-854c-c27d803f8e5b-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:34.772314 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:34.772280    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8efc126d-8902-4ce6-97b0-b4c47d8a645b" path="/var/lib/kubelet/pods/8efc126d-8902-4ce6-97b0-b4c47d8a645b/volumes"
Sep 16 01:05:35.022718 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:35.022646    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"
Sep 16 01:05:35.022718 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:35.022691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz" event={"ID":"d734a671-4deb-4030-854c-c27d803f8e5b","Type":"ContainerDied","Data":"be88da0577e065cd0839f56ba2f924bf8e5abe1d189ffbf31e7841ad992f86f5"}
Sep 16 01:05:35.022898 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:35.022753    2573 scope.go:117] "RemoveContainer" containerID="51b672f5dd423751f8412b42972cbfbda3f7734dbc11646e91c547d39d670d13"
Sep 16 01:05:35.041547 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:35.041503    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"]
Sep 16 01:05:35.044324 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:35.044302    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f097ba39-fdc9664fd-hjcxz"]
Sep 16 01:05:36.037946 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.037907    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"]
Sep 16 01:05:36.038486 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.038152    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6" podUID="64424d37-f681-416b-ac15-0616e5c064df" containerName="payload-processing" containerID="cri-o://d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982" gracePeriod=30
Sep 16 01:05:36.043689 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.043646    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"]
Sep 16 01:05:36.043892 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.043836    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g" podUID="8400e400-f206-4708-a85c-d64d5e8da8a0" containerName="payload-pre-processing" containerID="cri-o://2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8" gracePeriod=30
Sep 16 01:05:36.251143 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.251112    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"]
Sep 16 01:05:36.255222 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.255196    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d58169-49bqk"]
Sep 16 01:05:36.313573 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.313542    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:05:36.350980 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.350958    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:05:36.385370 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.385339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64424d37-f681-416b-ac15-0616e5c064df-plugins-config-volume\") pod \"64424d37-f681-416b-ac15-0616e5c064df\" (UID: \"64424d37-f681-416b-ac15-0616e5c064df\") "
Sep 16 01:05:36.385730 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.385469    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x75zp\" (UniqueName: \"kubernetes.io/projected/64424d37-f681-416b-ac15-0616e5c064df-kube-api-access-x75zp\") pod \"64424d37-f681-416b-ac15-0616e5c064df\" (UID: \"64424d37-f681-416b-ac15-0616e5c064df\") "
Sep 16 01:05:36.386967 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.386508    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64424d37-f681-416b-ac15-0616e5c064df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64424d37-f681-416b-ac15-0616e5c064df" (UID: "64424d37-f681-416b-ac15-0616e5c064df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:36.386967 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.386950    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64424d37-f681-416b-ac15-0616e5c064df-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:36.390410 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.390372    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64424d37-f681-416b-ac15-0616e5c064df-kube-api-access-x75zp" (OuterVolumeSpecName: "kube-api-access-x75zp") pod "64424d37-f681-416b-ac15-0616e5c064df" (UID: "64424d37-f681-416b-ac15-0616e5c064df"). InnerVolumeSpecName "kube-api-access-x75zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:36.487993 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.487957    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x8w2\" (UniqueName: \"kubernetes.io/projected/8400e400-f206-4708-a85c-d64d5e8da8a0-kube-api-access-8x8w2\") pod \"8400e400-f206-4708-a85c-d64d5e8da8a0\" (UID: \"8400e400-f206-4708-a85c-d64d5e8da8a0\") "
Sep 16 01:05:36.488187 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.488068    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8400e400-f206-4708-a85c-d64d5e8da8a0-plugins-config-volume\") pod \"8400e400-f206-4708-a85c-d64d5e8da8a0\" (UID: \"8400e400-f206-4708-a85c-d64d5e8da8a0\") "
Sep 16 01:05:36.488405 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.488381    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8400e400-f206-4708-a85c-d64d5e8da8a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8400e400-f206-4708-a85c-d64d5e8da8a0" (UID: "8400e400-f206-4708-a85c-d64d5e8da8a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:36.488492 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.488479    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8400e400-f206-4708-a85c-d64d5e8da8a0-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:36.488557 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.488494    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75zp\" (UniqueName: \"kubernetes.io/projected/64424d37-f681-416b-ac15-0616e5c064df-kube-api-access-x75zp\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:36.490172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.490150    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8400e400-f206-4708-a85c-d64d5e8da8a0-kube-api-access-8x8w2" (OuterVolumeSpecName: "kube-api-access-8x8w2") pod "8400e400-f206-4708-a85c-d64d5e8da8a0" (UID: "8400e400-f206-4708-a85c-d64d5e8da8a0"). InnerVolumeSpecName "kube-api-access-8x8w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:36.589785 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.589712    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x8w2\" (UniqueName: \"kubernetes.io/projected/8400e400-f206-4708-a85c-d64d5e8da8a0-kube-api-access-8x8w2\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:36.778919 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.778887    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="594ca788-897c-4cb6-8499-adcd63b2953e" path="/var/lib/kubelet/pods/594ca788-897c-4cb6-8499-adcd63b2953e/volumes"
Sep 16 01:05:36.779281 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:36.779267    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d734a671-4deb-4030-854c-c27d803f8e5b" path="/var/lib/kubelet/pods/d734a671-4deb-4030-854c-c27d803f8e5b/volumes"
Sep 16 01:05:37.040314 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.040280    2573 generic.go:356] "Generic (PLEG): container finished" podID="8400e400-f206-4708-a85c-d64d5e8da8a0" containerID="2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8" exitCode=0
Sep 16 01:05:37.040798 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.040345    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"
Sep 16 01:05:37.040798 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.040373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g" event={"ID":"8400e400-f206-4708-a85c-d64d5e8da8a0","Type":"ContainerDied","Data":"2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8"}
Sep 16 01:05:37.040798 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.040440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g" event={"ID":"8400e400-f206-4708-a85c-d64d5e8da8a0","Type":"ContainerDied","Data":"aaf54b4b4f4240129ddaeb648b8eb9db96c19277fc7e2246c2e56b7706da8679"}
Sep 16 01:05:37.040798 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.040476    2573 scope.go:117] "RemoveContainer" containerID="2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8"
Sep 16 01:05:37.041927 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.041906    2573 generic.go:356] "Generic (PLEG): container finished" podID="64424d37-f681-416b-ac15-0616e5c064df" containerID="d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982" exitCode=0
Sep 16 01:05:37.041996 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.041937    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6" event={"ID":"64424d37-f681-416b-ac15-0616e5c064df","Type":"ContainerDied","Data":"d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982"}
Sep 16 01:05:37.041996 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.041958    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"
Sep 16 01:05:37.041996 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.041973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6" event={"ID":"64424d37-f681-416b-ac15-0616e5c064df","Type":"ContainerDied","Data":"6ea5eafccf5a21ff82f55f7a319dd429ee13c4bd7b5ca42ac20ac68a13c5efe9"}
Sep 16 01:05:37.053087 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.053071    2573 scope.go:117] "RemoveContainer" containerID="2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8"
Sep 16 01:05:37.053373 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:05:37.053352    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8\": container with ID starting with 2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8 not found: ID does not exist" containerID="2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8"
Sep 16 01:05:37.053470 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.053380    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8"} err="failed to get container status \"2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8\": rpc error: code = NotFound desc = could not find container \"2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8\": container with ID starting with 2b34596d88dd672c8eab4e521b5368c075a38eef1fb4b7722410a005f9b21fb8 not found: ID does not exist"
Sep 16 01:05:37.053470 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.053396    2573 scope.go:117] "RemoveContainer" containerID="d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982"
Sep 16 01:05:37.061077 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.061059    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"]
Sep 16 01:05:37.064478 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.064457    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d58169-f98bd68d6-bzh2g"]
Sep 16 01:05:37.066962 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.066946    2573 scope.go:117] "RemoveContainer" containerID="d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982"
Sep 16 01:05:37.067208 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:05:37.067187    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982\": container with ID starting with d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982 not found: ID does not exist" containerID="d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982"
Sep 16 01:05:37.067272 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.067217    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982"} err="failed to get container status \"d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982\": rpc error: code = NotFound desc = could not find container \"d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982\": container with ID starting with d70503f464ed33a7674adde567dab8690af70ebbe4991b529278b3aea9412982 not found: ID does not exist"
Sep 16 01:05:37.075420 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.075398    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"]
Sep 16 01:05:37.077575 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:37.077498    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d58169-cb6c754ff-r9hv6"]
Sep 16 01:05:38.772284 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:38.772249    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64424d37-f681-416b-ac15-0616e5c064df" path="/var/lib/kubelet/pods/64424d37-f681-416b-ac15-0616e5c064df/volumes"
Sep 16 01:05:38.772664 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:38.772633    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8400e400-f206-4708-a85c-d64d5e8da8a0" path="/var/lib/kubelet/pods/8400e400-f206-4708-a85c-d64d5e8da8a0/volumes"
Sep 16 01:05:39.589828 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.589798    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"]
Sep 16 01:05:39.590236 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590224    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8400e400-f206-4708-a85c-d64d5e8da8a0" containerName="payload-pre-processing"
Sep 16 01:05:39.590277 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590237    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8400e400-f206-4708-a85c-d64d5e8da8a0" containerName="payload-pre-processing"
Sep 16 01:05:39.590277 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590260    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64424d37-f681-416b-ac15-0616e5c064df" containerName="payload-processing"
Sep 16 01:05:39.590277 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590266    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="64424d37-f681-416b-ac15-0616e5c064df" containerName="payload-processing"
Sep 16 01:05:39.590374 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590278    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d734a671-4deb-4030-854c-c27d803f8e5b" containerName="payload-pre-processing"
Sep 16 01:05:39.590374 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590284    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d734a671-4deb-4030-854c-c27d803f8e5b" containerName="payload-pre-processing"
Sep 16 01:05:39.590374 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590355    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="64424d37-f681-416b-ac15-0616e5c064df" containerName="payload-processing"
Sep 16 01:05:39.590374 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590361    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d734a671-4deb-4030-854c-c27d803f8e5b" containerName="payload-pre-processing"
Sep 16 01:05:39.590374 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.590370    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8400e400-f206-4708-a85c-d64d5e8da8a0" containerName="payload-pre-processing"
Sep 16 01:05:39.594868 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.594851    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:39.597113 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.597092    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a58c3d9e-dockercfg-mp872\""
Sep 16 01:05:39.597241 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.597126    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a58c3d9e\""
Sep 16 01:05:39.605920 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.605897    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"]
Sep 16 01:05:39.720892 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.720857    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtb9\" (UniqueName: \"kubernetes.io/projected/2b21a616-35fa-41e8-967c-79de582cba22-kube-api-access-cgtb9\") pod \"payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg\" (UID: \"2b21a616-35fa-41e8-967c-79de582cba22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:39.721031 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.720917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b21a616-35fa-41e8-967c-79de582cba22-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg\" (UID: \"2b21a616-35fa-41e8-967c-79de582cba22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:39.822576 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.822539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtb9\" (UniqueName: \"kubernetes.io/projected/2b21a616-35fa-41e8-967c-79de582cba22-kube-api-access-cgtb9\") pod \"payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg\" (UID: \"2b21a616-35fa-41e8-967c-79de582cba22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:39.822988 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.822618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b21a616-35fa-41e8-967c-79de582cba22-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg\" (UID: \"2b21a616-35fa-41e8-967c-79de582cba22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:39.823298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.823279    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b21a616-35fa-41e8-967c-79de582cba22-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg\" (UID: \"2b21a616-35fa-41e8-967c-79de582cba22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:39.830358 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.830329    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtb9\" (UniqueName: \"kubernetes.io/projected/2b21a616-35fa-41e8-967c-79de582cba22-kube-api-access-cgtb9\") pod \"payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg\" (UID: \"2b21a616-35fa-41e8-967c-79de582cba22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:39.906284 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:39.906199    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:40.182265 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:40.181853    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"]
Sep 16 01:05:41.078623 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:41.078589    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg" event={"ID":"2b21a616-35fa-41e8-967c-79de582cba22","Type":"ContainerStarted","Data":"5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519"}
Sep 16 01:05:41.079063 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:41.078661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg" event={"ID":"2b21a616-35fa-41e8-967c-79de582cba22","Type":"ContainerStarted","Data":"f36fdde6c9928cf7995273128ca298e7e9a5656dd3a6993363d25946bfb9d5eb"}
Sep 16 01:05:41.079063 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:41.078730    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:41.095747 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:41.095704    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg" podStartSLOduration=2.095691346 podStartE2EDuration="2.095691346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:05:41.093237458 +0000 UTC m=+1326.796059668" watchObservedRunningTime="2026-09-16 01:05:41.095691346 +0000 UTC m=+1326.798513541"
Sep 16 01:05:41.100763 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:41.100742    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:41.100854 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:41.100779    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:41.113301 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:41.113281    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:42.095995 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:42.095966    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:05:42.843489 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:42.843460    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"]
Sep 16 01:05:42.843746 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:42.843722    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy" containerID="cri-o://10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935" gracePeriod=30
Sep 16 01:05:43.211834 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.211801    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"]
Sep 16 01:05:43.215700 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.215677    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"
Sep 16 01:05:43.217954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.217936    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:05:43.218038 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.217939    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:05:43.223131 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.223106    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"]
Sep 16 01:05:43.360105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.360076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc250a-121d-4f3e-916a-55784c374464-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh\" (UID: \"2bdc250a-121d-4f3e-916a-55784c374464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"
Sep 16 01:05:43.461566 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.461509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc250a-121d-4f3e-916a-55784c374464-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh\" (UID: \"2bdc250a-121d-4f3e-916a-55784c374464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"
Sep 16 01:05:43.462138 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.462095    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc250a-121d-4f3e-916a-55784c374464-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh\" (UID: \"2bdc250a-121d-4f3e-916a-55784c374464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"
Sep 16 01:05:43.528507 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.528464    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"
Sep 16 01:05:43.810382 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:43.810360    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"]
Sep 16 01:05:43.812395 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:05:43.812369    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdc250a_121d_4f3e_916a_55784c374464.slice/crio-23412f991a3fc8bf0835c132b855f2a66fd4cee1fc9810df2f5edb5703493917 WatchSource:0}: Error finding container 23412f991a3fc8bf0835c132b855f2a66fd4cee1fc9810df2f5edb5703493917: Status 404 returned error can't find the container with id 23412f991a3fc8bf0835c132b855f2a66fd4cee1fc9810df2f5edb5703493917
Sep 16 01:05:44.097580 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:44.097487    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh" event={"ID":"2bdc250a-121d-4f3e-916a-55784c374464","Type":"ContainerStarted","Data":"23412f991a3fc8bf0835c132b855f2a66fd4cee1fc9810df2f5edb5703493917"}
Sep 16 01:05:45.103462 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:45.103427    2573 generic.go:356] "Generic (PLEG): container finished" podID="2bdc250a-121d-4f3e-916a-55784c374464" containerID="289eeeae77b824680e220b737d7a518e490a429766829e4032c297fb253b176d" exitCode=0
Sep 16 01:05:45.103835 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:45.103533    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh" event={"ID":"2bdc250a-121d-4f3e-916a-55784c374464","Type":"ContainerDied","Data":"289eeeae77b824680e220b737d7a518e490a429766829e4032c297fb253b176d"}
Sep 16 01:05:46.263362 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:46.262733    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"
Sep 16 01:05:46.390450 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:46.390361    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc250a-121d-4f3e-916a-55784c374464-maas-api-service-ca\") pod \"2bdc250a-121d-4f3e-916a-55784c374464\" (UID: \"2bdc250a-121d-4f3e-916a-55784c374464\") "
Sep 16 01:05:46.390744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:46.390722    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdc250a-121d-4f3e-916a-55784c374464-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bdc250a-121d-4f3e-916a-55784c374464" (UID: "2bdc250a-121d-4f3e-916a-55784c374464"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:46.390915 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:46.390902    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc250a-121d-4f3e-916a-55784c374464-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:47.116082 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:47.116005    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"
Sep 16 01:05:47.116082 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:47.116044    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh" event={"ID":"2bdc250a-121d-4f3e-916a-55784c374464","Type":"ContainerDied","Data":"23412f991a3fc8bf0835c132b855f2a66fd4cee1fc9810df2f5edb5703493917"}
Sep 16 01:05:47.116263 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:47.116088    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23412f991a3fc8bf0835c132b855f2a66fd4cee1fc9810df2f5edb5703493917"
Sep 16 01:05:48.109511 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.109454    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:05:48.123165 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.123142    2573 generic.go:356] "Generic (PLEG): container finished" podID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerID="10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935" exitCode=0
Sep 16 01:05:48.123281 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.123192    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"
Sep 16 01:05:48.123281 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.123232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" event={"ID":"c67c81bd-f337-446c-9bce-977dd0fb592a","Type":"ContainerDied","Data":"10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935"}
Sep 16 01:05:48.123362 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.123286    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx" event={"ID":"c67c81bd-f337-446c-9bce-977dd0fb592a","Type":"ContainerDied","Data":"81c190ca9bb6245ea60a256866a2330a677da4a9a291564fa495fa90b5816907"}
Sep 16 01:05:48.123362 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.123313    2573 scope.go:117] "RemoveContainer" containerID="10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935"
Sep 16 01:05:48.134750 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.134725    2573 scope.go:117] "RemoveContainer" containerID="58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2"
Sep 16 01:05:48.143601 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.143582    2573 scope.go:117] "RemoveContainer" containerID="10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935"
Sep 16 01:05:48.143922 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:05:48.143900    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935\": container with ID starting with 10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935 not found: ID does not exist" containerID="10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935"
Sep 16 01:05:48.143972 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.143934    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935"} err="failed to get container status \"10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935\": rpc error: code = NotFound desc = could not find container \"10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935\": container with ID starting with 10ce7d0fd1e546f9e7b31d1e854461d2233de38f646d632d7c2b597f80a9f935 not found: ID does not exist"
Sep 16 01:05:48.143972 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.143958    2573 scope.go:117] "RemoveContainer" containerID="58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2"
Sep 16 01:05:48.144263 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:05:48.144239    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2\": container with ID starting with 58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2 not found: ID does not exist" containerID="58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2"
Sep 16 01:05:48.144320 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.144269    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2"} err="failed to get container status \"58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2\": rpc error: code = NotFound desc = could not find container \"58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2\": container with ID starting with 58998512db70f7577840122b985c6eceb23e9b770117d9df181f30b1347332a2 not found: ID does not exist"
Sep 16 01:05:48.199243 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.199196    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l" podUID="8de64c3e-9138-4b4c-a004-8d314a6c3778" containerName="main" containerID="cri-o://c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db" gracePeriod=45
Sep 16 01:05:48.207840 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.207811    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qtrd\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-kube-api-access-6qtrd\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.207922 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.207903    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-credential-socket\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.207990 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.207966    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-envoy\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.208048 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.207990    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-data\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.208104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208052    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-token\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.208157 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208103    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c81bd-f337-446c-9bce-977dd0fb592a-istiod-ca-cert\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.208157 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208144    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-podinfo\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.208265 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208174    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-certs\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.208265 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208203    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-socket\") pod \"c67c81bd-f337-446c-9bce-977dd0fb592a\" (UID: \"c67c81bd-f337-446c-9bce-977dd0fb592a\") "
Sep 16 01:05:48.208265 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208210    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.208538 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208504    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67c81bd-f337-446c-9bce-977dd0fb592a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:48.208626 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208616    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.208924 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208899    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.209015 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.208952    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-data" (OuterVolumeSpecName: "istio-data") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.209311 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.209084    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.210485 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.210459    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.210683 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.210646    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-kube-api-access-6qtrd" (OuterVolumeSpecName: "kube-api-access-6qtrd") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "kube-api-access-6qtrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:48.211038 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.211019    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:05:48.211576 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.211503    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-token" (OuterVolumeSpecName: "istio-token") pod "c67c81bd-f337-446c-9bce-977dd0fb592a" (UID: "c67c81bd-f337-446c-9bce-977dd0fb592a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:48.244941 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:05:48.244920    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8de64c3e_9138_4b4c_a004_8d314a6c3778.slice/crio-c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 01:05:48.310066 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.310044    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.310066 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.310068    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.310209 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.310080    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.310209 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.310091    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c81bd-f337-446c-9bce-977dd0fb592a-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.310209 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.310102    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c81bd-f337-446c-9bce-977dd0fb592a-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.310209 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.310114    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.310209 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.310125    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c81bd-f337-446c-9bce-977dd0fb592a-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.310209 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.310136    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qtrd\" (UniqueName: \"kubernetes.io/projected/c67c81bd-f337-446c-9bce-977dd0fb592a-kube-api-access-6qtrd\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.445860 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.445838    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"]
Sep 16 01:05:48.451659 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.451633    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d58169-openshift-default-74df6c49bd-5rqrx"]
Sep 16 01:05:48.453048 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.453031    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:05:48.511507 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511483    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-home\") pod \"8de64c3e-9138-4b4c-a004-8d314a6c3778\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") "
Sep 16 01:05:48.511639 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511543    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-dshm\") pod \"8de64c3e-9138-4b4c-a004-8d314a6c3778\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") "
Sep 16 01:05:48.511639 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511631    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de64c3e-9138-4b4c-a004-8d314a6c3778-tls-certs\") pod \"8de64c3e-9138-4b4c-a004-8d314a6c3778\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") "
Sep 16 01:05:48.511714 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511660    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzfr7\" (UniqueName: \"kubernetes.io/projected/8de64c3e-9138-4b4c-a004-8d314a6c3778-kube-api-access-jzfr7\") pod \"8de64c3e-9138-4b4c-a004-8d314a6c3778\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") "
Sep 16 01:05:48.511714 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511687    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-model-cache\") pod \"8de64c3e-9138-4b4c-a004-8d314a6c3778\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") "
Sep 16 01:05:48.511714 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511702    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-tmp-dir\") pod \"8de64c3e-9138-4b4c-a004-8d314a6c3778\" (UID: \"8de64c3e-9138-4b4c-a004-8d314a6c3778\") "
Sep 16 01:05:48.511863 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511745    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-home" (OuterVolumeSpecName: "home") pod "8de64c3e-9138-4b4c-a004-8d314a6c3778" (UID: "8de64c3e-9138-4b4c-a004-8d314a6c3778"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.512026 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511975    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-model-cache" (OuterVolumeSpecName: "model-cache") pod "8de64c3e-9138-4b4c-a004-8d314a6c3778" (UID: "8de64c3e-9138-4b4c-a004-8d314a6c3778"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.512118 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.512019    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8de64c3e-9138-4b4c-a004-8d314a6c3778" (UID: "8de64c3e-9138-4b4c-a004-8d314a6c3778"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.512118 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.511991    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.513949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.513930    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-dshm" (OuterVolumeSpecName: "dshm") pod "8de64c3e-9138-4b4c-a004-8d314a6c3778" (UID: "8de64c3e-9138-4b4c-a004-8d314a6c3778"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:48.513949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.513937    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de64c3e-9138-4b4c-a004-8d314a6c3778-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8de64c3e-9138-4b4c-a004-8d314a6c3778" (UID: "8de64c3e-9138-4b4c-a004-8d314a6c3778"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:48.513949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.513937    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de64c3e-9138-4b4c-a004-8d314a6c3778-kube-api-access-jzfr7" (OuterVolumeSpecName: "kube-api-access-jzfr7") pod "8de64c3e-9138-4b4c-a004-8d314a6c3778" (UID: "8de64c3e-9138-4b4c-a004-8d314a6c3778"). InnerVolumeSpecName "kube-api-access-jzfr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:48.612614 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.612590    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.612614 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.612611    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de64c3e-9138-4b4c-a004-8d314a6c3778-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.612767 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.612622    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzfr7\" (UniqueName: \"kubernetes.io/projected/8de64c3e-9138-4b4c-a004-8d314a6c3778-kube-api-access-jzfr7\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.612767 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.612631    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.612767 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.612643    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de64c3e-9138-4b4c-a004-8d314a6c3778-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:05:48.771958 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:48.771924    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" path="/var/lib/kubelet/pods/c67c81bd-f337-446c-9bce-977dd0fb592a/volumes"
Sep 16 01:05:49.132347 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.132260    2573 generic.go:356] "Generic (PLEG): container finished" podID="8de64c3e-9138-4b4c-a004-8d314a6c3778" containerID="c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db" exitCode=0
Sep 16 01:05:49.132347 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.132311    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"
Sep 16 01:05:49.132839 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.132349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l" event={"ID":"8de64c3e-9138-4b4c-a004-8d314a6c3778","Type":"ContainerDied","Data":"c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db"}
Sep 16 01:05:49.132839 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.132395    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l" event={"ID":"8de64c3e-9138-4b4c-a004-8d314a6c3778","Type":"ContainerDied","Data":"d4007c18daf29cc6e71b3b0f60d6b601b0ceb2246a11d3c96b898cb24b6ddd72"}
Sep 16 01:05:49.132839 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.132418    2573 scope.go:117] "RemoveContainer" containerID="c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db"
Sep 16 01:05:49.141929 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.141910    2573 scope.go:117] "RemoveContainer" containerID="c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db"
Sep 16 01:05:49.142190 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:05:49.142164    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db\": container with ID starting with c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db not found: ID does not exist" containerID="c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db"
Sep 16 01:05:49.142331 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.142193    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db"} err="failed to get container status \"c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db\": rpc error: code = NotFound desc = could not find container \"c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db\": container with ID starting with c5e0ae5fbfedc902b888845c9c12c77440bb07a9a7f2a0e6a48f7077477602db not found: ID does not exist"
Sep 16 01:05:49.147792 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.147772    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"]
Sep 16 01:05:49.158157 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:49.152344    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6d0d72/test-model-6d0d72-kserve-77d846d67f-mbg5l"]
Sep 16 01:05:50.771127 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:50.771095    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de64c3e-9138-4b4c-a004-8d314a6c3778" path="/var/lib/kubelet/pods/8de64c3e-9138-4b4c-a004-8d314a6c3778/volumes"
Sep 16 01:05:52.087249 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:52.087222    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:05:53.967117 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:53.967081    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"]
Sep 16 01:05:53.972210 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:53.972175    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d0d72-qmwmh"]
Sep 16 01:05:54.771080 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:54.771050    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdc250a-121d-4f3e-916a-55784c374464" path="/var/lib/kubelet/pods/2bdc250a-121d-4f3e-916a-55784c374464/volumes"
Sep 16 01:05:57.769296 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769212    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"]
Sep 16 01:05:57.769873 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769852    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy"
Sep 16 01:05:57.769959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769876    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy"
Sep 16 01:05:57.769959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769908    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de64c3e-9138-4b4c-a004-8d314a6c3778" containerName="main"
Sep 16 01:05:57.769959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769916    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de64c3e-9138-4b4c-a004-8d314a6c3778" containerName="main"
Sep 16 01:05:57.769959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769933    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdc250a-121d-4f3e-916a-55784c374464" containerName="revoke-keys"
Sep 16 01:05:57.769959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769942    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdc250a-121d-4f3e-916a-55784c374464" containerName="revoke-keys"
Sep 16 01:05:57.769959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769953    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy"
Sep 16 01:05:57.770248 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.769962    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy"
Sep 16 01:05:57.770248 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.770072    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy"
Sep 16 01:05:57.770248 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.770083    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de64c3e-9138-4b4c-a004-8d314a6c3778" containerName="main"
Sep 16 01:05:57.770248 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.770095    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67c81bd-f337-446c-9bce-977dd0fb592a" containerName="istio-proxy"
Sep 16 01:05:57.770248 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.770106    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdc250a-121d-4f3e-916a-55784c374464" containerName="revoke-keys"
Sep 16 01:05:57.774614 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.774591    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"
Sep 16 01:05:57.777984 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.777957    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:05:57.778241 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.778220    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:05:57.779064 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.779041    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"]
Sep 16 01:05:57.898139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.898107    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7388c154-1f5e-4ddc-98fb-425015f453d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk\" (UID: \"7388c154-1f5e-4ddc-98fb-425015f453d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"
Sep 16 01:05:57.999487 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:57.999460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7388c154-1f5e-4ddc-98fb-425015f453d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk\" (UID: \"7388c154-1f5e-4ddc-98fb-425015f453d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"
Sep 16 01:05:58.000080 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:58.000061    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7388c154-1f5e-4ddc-98fb-425015f453d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk\" (UID: \"7388c154-1f5e-4ddc-98fb-425015f453d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"
Sep 16 01:05:58.087968 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:58.087904    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"
Sep 16 01:05:58.359881 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:58.359858    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"]
Sep 16 01:05:58.361829 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:05:58.361801    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7388c154_1f5e_4ddc_98fb_425015f453d4.slice/crio-8c88f7a51a8e8a0d37d0305ee38edaef9311e2088ee797fdb89a73f6da2155c5 WatchSource:0}: Error finding container 8c88f7a51a8e8a0d37d0305ee38edaef9311e2088ee797fdb89a73f6da2155c5: Status 404 returned error can't find the container with id 8c88f7a51a8e8a0d37d0305ee38edaef9311e2088ee797fdb89a73f6da2155c5
Sep 16 01:05:59.200698 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:59.200663    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk" event={"ID":"7388c154-1f5e-4ddc-98fb-425015f453d4","Type":"ContainerStarted","Data":"2ca4c3c9e6db43e328673693b8fc718506710c8821e9f3eab46dca0347c67f87"}
Sep 16 01:05:59.201075 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:59.200717    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk" event={"ID":"7388c154-1f5e-4ddc-98fb-425015f453d4","Type":"ContainerStarted","Data":"8c88f7a51a8e8a0d37d0305ee38edaef9311e2088ee797fdb89a73f6da2155c5"}
Sep 16 01:05:59.215815 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:05:59.215767    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk" podStartSLOduration=2.215753078 podStartE2EDuration="2.215753078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:05:59.213349476 +0000 UTC m=+1344.916171671" watchObservedRunningTime="2026-09-16 01:05:59.215753078 +0000 UTC m=+1344.918575297"
Sep 16 01:06:00.215406 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:00.215368    2573 generic.go:356] "Generic (PLEG): container finished" podID="7388c154-1f5e-4ddc-98fb-425015f453d4" containerID="2ca4c3c9e6db43e328673693b8fc718506710c8821e9f3eab46dca0347c67f87" exitCode=0
Sep 16 01:06:00.215778 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:00.215449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk" event={"ID":"7388c154-1f5e-4ddc-98fb-425015f453d4","Type":"ContainerDied","Data":"2ca4c3c9e6db43e328673693b8fc718506710c8821e9f3eab46dca0347c67f87"}
Sep 16 01:06:01.365412 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.365389    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"
Sep 16 01:06:01.430949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.428773    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"]
Sep 16 01:06:01.430949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.429455    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7388c154-1f5e-4ddc-98fb-425015f453d4" containerName="revoke-keys"
Sep 16 01:06:01.430949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.429476    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7388c154-1f5e-4ddc-98fb-425015f453d4" containerName="revoke-keys"
Sep 16 01:06:01.430949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.429589    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7388c154-1f5e-4ddc-98fb-425015f453d4" containerName="revoke-keys"
Sep 16 01:06:01.433437 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.433390    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.435754 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.435574    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7388c154-1f5e-4ddc-98fb-425015f453d4-maas-api-service-ca\") pod \"7388c154-1f5e-4ddc-98fb-425015f453d4\" (UID: \"7388c154-1f5e-4ddc-98fb-425015f453d4\") "
Sep 16 01:06:01.436051 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.436029    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7388c154-1f5e-4ddc-98fb-425015f453d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7388c154-1f5e-4ddc-98fb-425015f453d4" (UID: "7388c154-1f5e-4ddc-98fb-425015f453d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:06:01.436126 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.436071    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f831c5-openshift-default-dockercfg-zw8qk\""
Sep 16 01:06:01.442715 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.442684    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"]
Sep 16 01:06:01.536861 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.536758    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-data\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.536861 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.536813    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-socket\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.536861 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.536843    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-credential-socket\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.537155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.536864    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8459a3f-245f-42bd-ba92-29fc424c65fa-istiod-ca-cert\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.537155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.536900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-certs\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.537155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.536959    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-podinfo\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.537155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.537009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79d7\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-kube-api-access-z79d7\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.537155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.537066    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-envoy\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.537155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.537145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-token\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.537460 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.537203    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7388c154-1f5e-4ddc-98fb-425015f453d4-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:06:01.638931 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.638884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-envoy\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.638998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-token\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639055    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-data\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-socket\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-credential-socket\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639354 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639145    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8459a3f-245f-42bd-ba92-29fc424c65fa-istiod-ca-cert\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639354 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639177    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-certs\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639354 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-podinfo\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639354 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639298    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79d7\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-kube-api-access-z79d7\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639648 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639512    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-credential-socket\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639994 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639944    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-socket\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.639994 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.639977    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8459a3f-245f-42bd-ba92-29fc424c65fa-istiod-ca-cert\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.640176 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.640070    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-data\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.640236 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.640175    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-certs\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.642090 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.642067    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-envoy\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.642224 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.642206    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-podinfo\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.647312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.647288    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79d7\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-kube-api-access-z79d7\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.647403 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.647316    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-token\") pod \"e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") " pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:01.748804 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:01.748767    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:02.027258 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.027234    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"]
Sep 16 01:06:02.029028 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:06:02.028994    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8459a3f_245f_42bd_ba92_29fc424c65fa.slice/crio-e1936bc956835861e4e03053686ee1bb9e94534bb62693eff1094422e0cff2a0 WatchSource:0}: Error finding container e1936bc956835861e4e03053686ee1bb9e94534bb62693eff1094422e0cff2a0: Status 404 returned error can't find the container with id e1936bc956835861e4e03053686ee1bb9e94534bb62693eff1094422e0cff2a0
Sep 16 01:06:02.032113 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.032072    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:02.032206 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.032160    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:02.032265 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.032203    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:02.229476 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.229446    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"
Sep 16 01:06:02.229682 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.229453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk" event={"ID":"7388c154-1f5e-4ddc-98fb-425015f453d4","Type":"ContainerDied","Data":"8c88f7a51a8e8a0d37d0305ee38edaef9311e2088ee797fdb89a73f6da2155c5"}
Sep 16 01:06:02.229682 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.229617    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c88f7a51a8e8a0d37d0305ee38edaef9311e2088ee797fdb89a73f6da2155c5"
Sep 16 01:06:02.231599 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.231574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" event={"ID":"f8459a3f-245f-42bd-ba92-29fc424c65fa","Type":"ContainerStarted","Data":"beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3"}
Sep 16 01:06:02.231726 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.231620    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" event={"ID":"f8459a3f-245f-42bd-ba92-29fc424c65fa","Type":"ContainerStarted","Data":"e1936bc956835861e4e03053686ee1bb9e94534bb62693eff1094422e0cff2a0"}
Sep 16 01:06:02.253700 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.253639    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" podStartSLOduration=1.253619841 podStartE2EDuration="1.253619841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:02.250715691 +0000 UTC m=+1347.953537887" watchObservedRunningTime="2026-09-16 01:06:02.253619841 +0000 UTC m=+1347.956442036"
Sep 16 01:06:02.749125 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:02.749093    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:03.755660 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:03.755633    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:04.243657 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:04.243547    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:04.271249 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:04.244467    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:07.305410 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.305374    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"]
Sep 16 01:06:07.309288 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.309265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:07.312623 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.312600    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f831c5\""
Sep 16 01:06:07.312861 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.312845    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f831c5-dockercfg-4sw4k\""
Sep 16 01:06:07.318547 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.318504    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"]
Sep 16 01:06:07.401588 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.401552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4\" (UID: \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:07.401828 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.401663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmr4p\" (UniqueName: \"kubernetes.io/projected/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-kube-api-access-pmr4p\") pod \"payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4\" (UID: \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:07.503155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.503119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4\" (UID: \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:07.503301 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.503179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr4p\" (UniqueName: \"kubernetes.io/projected/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-kube-api-access-pmr4p\") pod \"payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4\" (UID: \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:07.503847 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.503827    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4\" (UID: \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:07.510797 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.510777    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmr4p\" (UniqueName: \"kubernetes.io/projected/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-kube-api-access-pmr4p\") pod \"payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4\" (UID: \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:07.623652 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.623582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:07.902542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:07.902500    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"]
Sep 16 01:06:07.904595 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:06:07.904570    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe07b1e_fe22_40a8_a367_abef6b6adaf0.slice/crio-116f3fb70a32adefcbc53d9f315f49537fa1344af4e10ae9a43807cbad8642ab WatchSource:0}: Error finding container 116f3fb70a32adefcbc53d9f315f49537fa1344af4e10ae9a43807cbad8642ab: Status 404 returned error can't find the container with id 116f3fb70a32adefcbc53d9f315f49537fa1344af4e10ae9a43807cbad8642ab
Sep 16 01:06:08.268328 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:08.268295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4" event={"ID":"6fe07b1e-fe22-40a8-a367-abef6b6adaf0","Type":"ContainerStarted","Data":"268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a"}
Sep 16 01:06:08.268515 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:08.268355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4" event={"ID":"6fe07b1e-fe22-40a8-a367-abef6b6adaf0","Type":"ContainerStarted","Data":"116f3fb70a32adefcbc53d9f315f49537fa1344af4e10ae9a43807cbad8642ab"}
Sep 16 01:06:08.268515 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:08.268385    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:08.284114 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:08.284060    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4" podStartSLOduration=1.2840409400000001 podStartE2EDuration="1.28404094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:08.282295644 +0000 UTC m=+1353.985117840" watchObservedRunningTime="2026-09-16 01:06:08.28404094 +0000 UTC m=+1353.986863140"
Sep 16 01:06:09.176892 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.176856    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"]
Sep 16 01:06:09.177492 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.177461    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg" podUID="2b21a616-35fa-41e8-967c-79de582cba22" containerName="payload-pre-processing" containerID="cri-o://5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519" gracePeriod=30
Sep 16 01:06:09.367957 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.367927    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"]
Sep 16 01:06:09.371746 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.371717    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a58c3d9e-xzgxk"]
Sep 16 01:06:09.438142 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.438077    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:06:09.530207 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.530178    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b21a616-35fa-41e8-967c-79de582cba22-plugins-config-volume\") pod \"2b21a616-35fa-41e8-967c-79de582cba22\" (UID: \"2b21a616-35fa-41e8-967c-79de582cba22\") "
Sep 16 01:06:09.530387 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.530236    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgtb9\" (UniqueName: \"kubernetes.io/projected/2b21a616-35fa-41e8-967c-79de582cba22-kube-api-access-cgtb9\") pod \"2b21a616-35fa-41e8-967c-79de582cba22\" (UID: \"2b21a616-35fa-41e8-967c-79de582cba22\") "
Sep 16 01:06:09.532255 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.531382    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b21a616-35fa-41e8-967c-79de582cba22-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b21a616-35fa-41e8-967c-79de582cba22" (UID: "2b21a616-35fa-41e8-967c-79de582cba22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:06:09.533554 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.532855    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b21a616-35fa-41e8-967c-79de582cba22-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:06:09.534101 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.534071    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b21a616-35fa-41e8-967c-79de582cba22-kube-api-access-cgtb9" (OuterVolumeSpecName: "kube-api-access-cgtb9") pod "2b21a616-35fa-41e8-967c-79de582cba22" (UID: "2b21a616-35fa-41e8-967c-79de582cba22"). InnerVolumeSpecName "kube-api-access-cgtb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:06:09.634623 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:09.634539    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtb9\" (UniqueName: \"kubernetes.io/projected/2b21a616-35fa-41e8-967c-79de582cba22-kube-api-access-cgtb9\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:06:10.281841 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.281809    2573 generic.go:356] "Generic (PLEG): container finished" podID="2b21a616-35fa-41e8-967c-79de582cba22" containerID="5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519" exitCode=0
Sep 16 01:06:10.282333 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.281879    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"
Sep 16 01:06:10.282333 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.281905    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg" event={"ID":"2b21a616-35fa-41e8-967c-79de582cba22","Type":"ContainerDied","Data":"5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519"}
Sep 16 01:06:10.282333 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.281962    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg" event={"ID":"2b21a616-35fa-41e8-967c-79de582cba22","Type":"ContainerDied","Data":"f36fdde6c9928cf7995273128ca298e7e9a5656dd3a6993363d25946bfb9d5eb"}
Sep 16 01:06:10.282333 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.281986    2573 scope.go:117] "RemoveContainer" containerID="5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519"
Sep 16 01:06:10.292256 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.292243    2573 scope.go:117] "RemoveContainer" containerID="5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519"
Sep 16 01:06:10.292545 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:06:10.292505    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519\": container with ID starting with 5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519 not found: ID does not exist" containerID="5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519"
Sep 16 01:06:10.292594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.292554    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519"} err="failed to get container status \"5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519\": rpc error: code = NotFound desc = could not find container \"5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519\": container with ID starting with 5a89a32af394506b4f882d6cf5f1bea289185ef41121da835393df01ecf24519 not found: ID does not exist"
Sep 16 01:06:10.310997 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.310963    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"]
Sep 16 01:06:10.313389 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.313369    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a58c3d9e-6f8bfd97c4-ltdgg"]
Sep 16 01:06:10.772111 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.772079    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b21a616-35fa-41e8-967c-79de582cba22" path="/var/lib/kubelet/pods/2b21a616-35fa-41e8-967c-79de582cba22/volumes"
Sep 16 01:06:10.772427 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:10.772415    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7388c154-1f5e-4ddc-98fb-425015f453d4" path="/var/lib/kubelet/pods/7388c154-1f5e-4ddc-98fb-425015f453d4/volumes"
Sep 16 01:06:17.764957 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:17.764908    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:06:19.277336 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.277306    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:06:19.637688 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.637611    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"]
Sep 16 01:06:19.638311 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.638292    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b21a616-35fa-41e8-967c-79de582cba22" containerName="payload-pre-processing"
Sep 16 01:06:19.638311 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.638312    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b21a616-35fa-41e8-967c-79de582cba22" containerName="payload-pre-processing"
Sep 16 01:06:19.638465 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.638403    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b21a616-35fa-41e8-967c-79de582cba22" containerName="payload-pre-processing"
Sep 16 01:06:19.642995 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.642973    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.645289 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.645269    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-0ee1bf-openshift-default-dockercfg-cd8gf\""
Sep 16 01:06:19.653785 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.653764    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"]
Sep 16 01:06:19.837608 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.837569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-envoy\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.837799 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.837631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-token\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.837799 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.837666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpbl\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-kube-api-access-8tpbl\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.837799 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.837788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-data\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.837959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.837848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-certs\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.837959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.837897    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-socket\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.838051 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.837953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-credential-socket\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.838051 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.837987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.838167 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.838055    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-podinfo\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.939000 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.938894    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-socket\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.939280 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.939258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-credential-socket\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.939795 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.939775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.940649 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.940630    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-podinfo\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.941331 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.941305    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-envoy\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.941509 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.941344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-token\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.941509 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.941387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpbl\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-kube-api-access-8tpbl\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.941509 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.941478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-data\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.941717 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.941555    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-certs\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.941885 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.941852    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-certs\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.941956 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.939495    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-socket\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.942112 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.939712    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-credential-socket\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.942112 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.940553    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.942907 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.942873    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-data\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.945720 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.945700    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-podinfo\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.946339 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.946315    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-envoy\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.950007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.949986    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-token\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.950123 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.950106    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpbl\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-kube-api-access-8tpbl\") pod \"e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") " pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:19.955941 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:19.955919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:20.241547 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:20.237952    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"]
Sep 16 01:06:20.248593 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:06:20.248561    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf554aabb_aac0_4371_9934_1cdf2ea7d6ba.slice/crio-3d6a6994537d61975d9a6acefe0efecd556e237221fd70dfbcf010437ab5d65c WatchSource:0}: Error finding container 3d6a6994537d61975d9a6acefe0efecd556e237221fd70dfbcf010437ab5d65c: Status 404 returned error can't find the container with id 3d6a6994537d61975d9a6acefe0efecd556e237221fd70dfbcf010437ab5d65c
Sep 16 01:06:20.251115 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:20.251085    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:20.251188 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:20.251147    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:20.251188 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:20.251176    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:20.347198 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:20.347153    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" event={"ID":"f554aabb-aac0-4371-9934-1cdf2ea7d6ba","Type":"ContainerStarted","Data":"336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433"}
Sep 16 01:06:20.347671 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:20.347218    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" event={"ID":"f554aabb-aac0-4371-9934-1cdf2ea7d6ba","Type":"ContainerStarted","Data":"3d6a6994537d61975d9a6acefe0efecd556e237221fd70dfbcf010437ab5d65c"}
Sep 16 01:06:20.364915 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:20.364849    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" podStartSLOduration=1.3648291399999999 podStartE2EDuration="1.36482914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:20.364322084 +0000 UTC m=+1366.067144279" watchObservedRunningTime="2026-09-16 01:06:20.36482914 +0000 UTC m=+1366.067651338"
Sep 16 01:06:20.956451 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:20.956423    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:21.962402 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:21.962372    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:22.359054 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:22.358965    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:22.359971 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:22.359950    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:06:40.474649 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:40.474617    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerID="beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3" exitCode=0
Sep 16 01:06:40.475047 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:40.474686    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" event={"ID":"f8459a3f-245f-42bd-ba92-29fc424c65fa","Type":"ContainerDied","Data":"beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3"}
Sep 16 01:06:40.475158 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:40.475141    2573 scope.go:117] "RemoveContainer" containerID="beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3"
Sep 16 01:06:40.476320 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:40.476288    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:40.476376 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:40.476363    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:40.476412 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:40.476399    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:06:41.482958 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:41.482923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" event={"ID":"f8459a3f-245f-42bd-ba92-29fc424c65fa","Type":"ContainerStarted","Data":"4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9"}
Sep 16 01:06:41.749994 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:41.749891    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:41.750176 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:41.750022    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 01:06:41.750176 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:41.750077    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 01:06:42.749485 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:42.749067    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 01:06:42.749485 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:42.749125    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 01:06:43.749319 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:43.749276    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 01:06:43.749538 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:43.749356    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 01:06:43.764804 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:43.764773    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:06:43.898495 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:43.898459    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"]
Sep 16 01:06:43.901449 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:43.901432    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"
Sep 16 01:06:43.903758 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:43.903732    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:06:43.903758 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:43.903733    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:06:43.909063 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:43.908816    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"]
Sep 16 01:06:44.020334 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:44.020257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe68395-bc99-4fd1-b184-d22625f369a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30fb702c-xgf52\" (UID: \"5fe68395-bc99-4fd1-b184-d22625f369a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"
Sep 16 01:06:44.121017 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:44.120985    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe68395-bc99-4fd1-b184-d22625f369a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30fb702c-xgf52\" (UID: \"5fe68395-bc99-4fd1-b184-d22625f369a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"
Sep 16 01:06:44.121597 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:44.121577    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe68395-bc99-4fd1-b184-d22625f369a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30fb702c-xgf52\" (UID: \"5fe68395-bc99-4fd1-b184-d22625f369a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"
Sep 16 01:06:44.214041 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:44.214008    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"
Sep 16 01:06:44.501258 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:44.501215    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"]
Sep 16 01:06:44.503176 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:06:44.503129    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe68395_bc99_4fd1_b184_d22625f369a4.slice/crio-84528caae8ae8e3ead449f606cbc974b0bbb596074d0dc9d85018dfe95cf59f5 WatchSource:0}: Error finding container 84528caae8ae8e3ead449f606cbc974b0bbb596074d0dc9d85018dfe95cf59f5: Status 404 returned error can't find the container with id 84528caae8ae8e3ead449f606cbc974b0bbb596074d0dc9d85018dfe95cf59f5
Sep 16 01:06:44.749631 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:44.749593    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 01:06:44.750091 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:44.749658    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 01:06:45.514590 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:45.514547    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52" event={"ID":"5fe68395-bc99-4fd1-b184-d22625f369a4","Type":"ContainerStarted","Data":"f81a0993c84888d1289d1588b4fe7b15d2c43f6bfde3b9c42061ef2571d84dfc"}
Sep 16 01:06:45.514801 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:45.514604    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52" event={"ID":"5fe68395-bc99-4fd1-b184-d22625f369a4","Type":"ContainerStarted","Data":"84528caae8ae8e3ead449f606cbc974b0bbb596074d0dc9d85018dfe95cf59f5"}
Sep 16 01:06:45.530674 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:45.530617    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52" podStartSLOduration=2.530596574 podStartE2EDuration="2.530596574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:45.527295046 +0000 UTC m=+1391.230117241" watchObservedRunningTime="2026-09-16 01:06:45.530596574 +0000 UTC m=+1391.233418770"
Sep 16 01:06:45.749447 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:45.749404    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 01:06:45.749947 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:45.749499    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 01:06:46.522941 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:46.522895    2573 generic.go:356] "Generic (PLEG): container finished" podID="5fe68395-bc99-4fd1-b184-d22625f369a4" containerID="f81a0993c84888d1289d1588b4fe7b15d2c43f6bfde3b9c42061ef2571d84dfc" exitCode=0
Sep 16 01:06:46.522941 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:46.522930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52" event={"ID":"5fe68395-bc99-4fd1-b184-d22625f369a4","Type":"ContainerDied","Data":"f81a0993c84888d1289d1588b4fe7b15d2c43f6bfde3b9c42061ef2571d84dfc"}
Sep 16 01:06:46.754225 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:46.754198    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:46.754658 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:46.754552    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:46.755317 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:46.755301    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:06:47.673868 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:47.673842    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"
Sep 16 01:06:47.765102 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:47.765072    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe68395-bc99-4fd1-b184-d22625f369a4-maas-api-service-ca\") pod \"5fe68395-bc99-4fd1-b184-d22625f369a4\" (UID: \"5fe68395-bc99-4fd1-b184-d22625f369a4\") "
Sep 16 01:06:47.765451 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:47.765399    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe68395-bc99-4fd1-b184-d22625f369a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fe68395-bc99-4fd1-b184-d22625f369a4" (UID: "5fe68395-bc99-4fd1-b184-d22625f369a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:06:47.866959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:47.866887    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe68395-bc99-4fd1-b184-d22625f369a4-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:06:48.538363 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:48.538328    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"
Sep 16 01:06:48.538590 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:48.538359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52" event={"ID":"5fe68395-bc99-4fd1-b184-d22625f369a4","Type":"ContainerDied","Data":"84528caae8ae8e3ead449f606cbc974b0bbb596074d0dc9d85018dfe95cf59f5"}
Sep 16 01:06:48.538590 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:48.538432    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84528caae8ae8e3ead449f606cbc974b0bbb596074d0dc9d85018dfe95cf59f5"
Sep 16 01:06:55.483187 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:55.483149    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"]
Sep 16 01:06:55.488560 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:55.488536    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fb702c-xgf52"]
Sep 16 01:06:56.776631 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:06:56.776598    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe68395-bc99-4fd1-b184-d22625f369a4" path="/var/lib/kubelet/pods/5fe68395-bc99-4fd1-b184-d22625f369a4/volumes"
Sep 16 01:07:04.640934 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:04.640902    2573 generic.go:356] "Generic (PLEG): container finished" podID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerID="336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433" exitCode=0
Sep 16 01:07:04.641365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:04.640978    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" event={"ID":"f554aabb-aac0-4371-9934-1cdf2ea7d6ba","Type":"ContainerDied","Data":"336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433"}
Sep 16 01:07:04.641424 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:04.641386    2573 scope.go:117] "RemoveContainer" containerID="336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433"
Sep 16 01:07:04.642512 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:04.642482    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:07:04.642606 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:04.642579    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:07:04.642646 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:04.642612    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:07:05.655319 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:05.654328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" event={"ID":"f554aabb-aac0-4371-9934-1cdf2ea7d6ba","Type":"ContainerStarted","Data":"28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d"}
Sep 16 01:07:05.956754 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:05.956704    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:07:05.957867 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:05.957835    2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 01:07:05.958021 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:05.957916    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 01:07:06.957182 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:06.957144    2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 01:07:06.957642 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:06.957230    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 01:07:07.359407 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:07.359328    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:07:07.956683 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:07.956642    2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 01:07:07.956867 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:07.956715    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 01:07:08.956239 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:08.956203    2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 01:07:08.956857 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:08.956815    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 01:07:09.956836 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:09.956796    2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 01:07:09.957458 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:09.957423    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 01:07:10.961661 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:10.961626    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:07:10.962623 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:10.962603    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:07:21.764996 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:21.764968    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:07:25.174088 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.174048    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"]
Sep 16 01:07:25.174552 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.174533    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe68395-bc99-4fd1-b184-d22625f369a4" containerName="revoke-keys"
Sep 16 01:07:25.174552 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.174548    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe68395-bc99-4fd1-b184-d22625f369a4" containerName="revoke-keys"
Sep 16 01:07:25.174647 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.174623    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe68395-bc99-4fd1-b184-d22625f369a4" containerName="revoke-keys"
Sep 16 01:07:25.178110 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.178095    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"
Sep 16 01:07:25.181455 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.181421    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:07:25.181973 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.181931    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:07:25.185322 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.185297    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"]
Sep 16 01:07:25.273693 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.273659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be0127-77e8-4656-b12f-ee51c5e154f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9\" (UID: \"c1be0127-77e8-4656-b12f-ee51c5e154f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"
Sep 16 01:07:25.374573 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.374509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be0127-77e8-4656-b12f-ee51c5e154f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9\" (UID: \"c1be0127-77e8-4656-b12f-ee51c5e154f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"
Sep 16 01:07:25.375261 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.375240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be0127-77e8-4656-b12f-ee51c5e154f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9\" (UID: \"c1be0127-77e8-4656-b12f-ee51c5e154f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"
Sep 16 01:07:25.492433 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.492388    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"
Sep 16 01:07:25.773920 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:25.773895    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"]
Sep 16 01:07:26.784932 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:26.784903    2573 generic.go:356] "Generic (PLEG): container finished" podID="c1be0127-77e8-4656-b12f-ee51c5e154f0" containerID="c8c6b33044ff96943c9436fa5fbe547acc1f9268cfaa96c3e999f44188cf9afc" exitCode=0
Sep 16 01:07:26.785301 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:26.784994    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9" event={"ID":"c1be0127-77e8-4656-b12f-ee51c5e154f0","Type":"ContainerDied","Data":"c8c6b33044ff96943c9436fa5fbe547acc1f9268cfaa96c3e999f44188cf9afc"}
Sep 16 01:07:26.785301 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:26.785047    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9" event={"ID":"c1be0127-77e8-4656-b12f-ee51c5e154f0","Type":"ContainerStarted","Data":"2bd50e171deae830f86d02ce3a2740c155201fb2382547eb61e9c874d0737e69"}
Sep 16 01:07:27.942872 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:27.942847    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"
Sep 16 01:07:28.003323 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:28.003293    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be0127-77e8-4656-b12f-ee51c5e154f0-maas-api-service-ca\") pod \"c1be0127-77e8-4656-b12f-ee51c5e154f0\" (UID: \"c1be0127-77e8-4656-b12f-ee51c5e154f0\") "
Sep 16 01:07:28.003662 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:28.003640    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1be0127-77e8-4656-b12f-ee51c5e154f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1be0127-77e8-4656-b12f-ee51c5e154f0" (UID: "c1be0127-77e8-4656-b12f-ee51c5e154f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:07:28.104352 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:28.104273    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be0127-77e8-4656-b12f-ee51c5e154f0-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:28.798934 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:28.798909    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"
Sep 16 01:07:28.799097 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:28.798902    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9" event={"ID":"c1be0127-77e8-4656-b12f-ee51c5e154f0","Type":"ContainerDied","Data":"2bd50e171deae830f86d02ce3a2740c155201fb2382547eb61e9c874d0737e69"}
Sep 16 01:07:28.799097 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:28.799023    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd50e171deae830f86d02ce3a2740c155201fb2382547eb61e9c874d0737e69"
Sep 16 01:07:30.745822 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.745779    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"]
Sep 16 01:07:30.746313 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.746252    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1be0127-77e8-4656-b12f-ee51c5e154f0" containerName="revoke-keys"
Sep 16 01:07:30.746313 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.746268    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1be0127-77e8-4656-b12f-ee51c5e154f0" containerName="revoke-keys"
Sep 16 01:07:30.746438 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.746367    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1be0127-77e8-4656-b12f-ee51c5e154f0" containerName="revoke-keys"
Sep 16 01:07:30.749838 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.749821    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"
Sep 16 01:07:30.751892 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.751871    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:07:30.751981 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.751916    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:07:30.758097 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.757822    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"]
Sep 16 01:07:30.831261 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.831224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed49c0-93dd-4a19-95b4-9a53754e32ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c\" (UID: \"91ed49c0-93dd-4a19-95b4-9a53754e32ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"
Sep 16 01:07:30.933323 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.933289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed49c0-93dd-4a19-95b4-9a53754e32ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c\" (UID: \"91ed49c0-93dd-4a19-95b4-9a53754e32ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"
Sep 16 01:07:30.933970 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:30.933949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed49c0-93dd-4a19-95b4-9a53754e32ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c\" (UID: \"91ed49c0-93dd-4a19-95b4-9a53754e32ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"
Sep 16 01:07:31.062725 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:31.062639    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"
Sep 16 01:07:31.341320 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:31.341245    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"]
Sep 16 01:07:31.344913 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:07:31.344882    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ed49c0_93dd_4a19_95b4_9a53754e32ab.slice/crio-acf352dc91f18f1da6f4576df4f24955664241e6dd4770779252260250b276c0 WatchSource:0}: Error finding container acf352dc91f18f1da6f4576df4f24955664241e6dd4770779252260250b276c0: Status 404 returned error can't find the container with id acf352dc91f18f1da6f4576df4f24955664241e6dd4770779252260250b276c0
Sep 16 01:07:31.817699 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:31.817660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c" event={"ID":"91ed49c0-93dd-4a19-95b4-9a53754e32ab","Type":"ContainerStarted","Data":"aa0c835e989b6c573138dc21fd05d628383d4b85777132aea7b91477d1f6ffb0"}
Sep 16 01:07:31.818161 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:31.817718    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c" event={"ID":"91ed49c0-93dd-4a19-95b4-9a53754e32ab","Type":"ContainerStarted","Data":"acf352dc91f18f1da6f4576df4f24955664241e6dd4770779252260250b276c0"}
Sep 16 01:07:31.835910 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:31.835855    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c" podStartSLOduration=1.835839892 podStartE2EDuration="1.835839892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:07:31.832978798 +0000 UTC m=+1437.535801010" watchObservedRunningTime="2026-09-16 01:07:31.835839892 +0000 UTC m=+1437.538662086"
Sep 16 01:07:31.862474 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:31.862439    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"]
Sep 16 01:07:31.864831 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:31.864811    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f50fc84-zpqh9"]
Sep 16 01:07:32.772021 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:32.771994    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1be0127-77e8-4656-b12f-ee51c5e154f0" path="/var/lib/kubelet/pods/c1be0127-77e8-4656-b12f-ee51c5e154f0/volumes"
Sep 16 01:07:32.824654 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:32.824611    2573 generic.go:356] "Generic (PLEG): container finished" podID="91ed49c0-93dd-4a19-95b4-9a53754e32ab" containerID="aa0c835e989b6c573138dc21fd05d628383d4b85777132aea7b91477d1f6ffb0" exitCode=0
Sep 16 01:07:32.825058 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:32.824663    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c" event={"ID":"91ed49c0-93dd-4a19-95b4-9a53754e32ab","Type":"ContainerDied","Data":"aa0c835e989b6c573138dc21fd05d628383d4b85777132aea7b91477d1f6ffb0"}
Sep 16 01:07:33.981763 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:33.981737    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"
Sep 16 01:07:34.067670 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:34.067635    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed49c0-93dd-4a19-95b4-9a53754e32ab-maas-api-service-ca\") pod \"91ed49c0-93dd-4a19-95b4-9a53754e32ab\" (UID: \"91ed49c0-93dd-4a19-95b4-9a53754e32ab\") "
Sep 16 01:07:34.068023 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:34.068000    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ed49c0-93dd-4a19-95b4-9a53754e32ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91ed49c0-93dd-4a19-95b4-9a53754e32ab" (UID: "91ed49c0-93dd-4a19-95b4-9a53754e32ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:07:34.169501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:34.169433    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed49c0-93dd-4a19-95b4-9a53754e32ab-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:34.840373 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:34.840345    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"
Sep 16 01:07:34.840543 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:34.840395    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c" event={"ID":"91ed49c0-93dd-4a19-95b4-9a53754e32ab","Type":"ContainerDied","Data":"acf352dc91f18f1da6f4576df4f24955664241e6dd4770779252260250b276c0"}
Sep 16 01:07:34.840543 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:34.840456    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf352dc91f18f1da6f4576df4f24955664241e6dd4770779252260250b276c0"
Sep 16 01:07:35.609057 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:35.609016    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"]
Sep 16 01:07:35.632683 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:35.632654    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"]
Sep 16 01:07:46.655310 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.655270    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"]
Sep 16 01:07:46.655930 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.655596    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47" podUID="26927ad2-6d42-4ab7-bde1-6d363b14afcd" containerName="payload-processing" containerID="cri-o://a85c6542551d9bd852d010bc78644103c54d677e75ac03d43738c989aff05362" gracePeriod=30
Sep 16 01:07:46.826938 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.826900    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"]
Sep 16 01:07:46.830354 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.830326    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2534e3-qj46c"]
Sep 16 01:07:46.917413 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.917354    2573 generic.go:356] "Generic (PLEG): container finished" podID="26927ad2-6d42-4ab7-bde1-6d363b14afcd" containerID="a85c6542551d9bd852d010bc78644103c54d677e75ac03d43738c989aff05362" exitCode=0
Sep 16 01:07:46.917548 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.917431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47" event={"ID":"26927ad2-6d42-4ab7-bde1-6d363b14afcd","Type":"ContainerDied","Data":"a85c6542551d9bd852d010bc78644103c54d677e75ac03d43738c989aff05362"}
Sep 16 01:07:46.917548 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.917494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47" event={"ID":"26927ad2-6d42-4ab7-bde1-6d363b14afcd","Type":"ContainerDied","Data":"cea6bd38e9e13001a6c48fb1de5de9530ffddd1ad11613ec80128028f6bd97cb"}
Sep 16 01:07:46.917548 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.917509    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cea6bd38e9e13001a6c48fb1de5de9530ffddd1ad11613ec80128028f6bd97cb"
Sep 16 01:07:46.921215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:46.921198    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:07:47.017060 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.017023    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26927ad2-6d42-4ab7-bde1-6d363b14afcd-plugins-config-volume\") pod \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\" (UID: \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\") "
Sep 16 01:07:47.017251 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.017096    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/26927ad2-6d42-4ab7-bde1-6d363b14afcd-kube-api-access-zv2s8\") pod \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\" (UID: \"26927ad2-6d42-4ab7-bde1-6d363b14afcd\") "
Sep 16 01:07:47.017470 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.017442    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26927ad2-6d42-4ab7-bde1-6d363b14afcd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26927ad2-6d42-4ab7-bde1-6d363b14afcd" (UID: "26927ad2-6d42-4ab7-bde1-6d363b14afcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:07:47.019325 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.019292    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26927ad2-6d42-4ab7-bde1-6d363b14afcd-kube-api-access-zv2s8" (OuterVolumeSpecName: "kube-api-access-zv2s8") pod "26927ad2-6d42-4ab7-bde1-6d363b14afcd" (UID: "26927ad2-6d42-4ab7-bde1-6d363b14afcd"). InnerVolumeSpecName "kube-api-access-zv2s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:47.118015 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.117976    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/26927ad2-6d42-4ab7-bde1-6d363b14afcd-kube-api-access-zv2s8\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:47.118015 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.118006    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26927ad2-6d42-4ab7-bde1-6d363b14afcd-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:47.923641 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.923614    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"
Sep 16 01:07:47.947564 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.947511    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"]
Sep 16 01:07:47.950448 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:47.950424    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2534e3-55c6dd8485-4lb47"]
Sep 16 01:07:48.771699 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:48.771670    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26927ad2-6d42-4ab7-bde1-6d363b14afcd" path="/var/lib/kubelet/pods/26927ad2-6d42-4ab7-bde1-6d363b14afcd/volumes"
Sep 16 01:07:48.772099 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:48.772080    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ed49c0-93dd-4a19-95b4-9a53754e32ab" path="/var/lib/kubelet/pods/91ed49c0-93dd-4a19-95b4-9a53754e32ab/volumes"
Sep 16 01:07:50.629654 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.629581    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46" podUID="2e74ae40-4e63-49f0-a734-d41735891c52" containerName="main" containerID="cri-o://eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5" gracePeriod=45
Sep 16 01:07:50.652292 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.652260    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh" podUID="00dd3601-90c7-4ac2-94a5-115d870b78ee" containerName="main" containerID="cri-o://d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5" gracePeriod=45
Sep 16 01:07:50.907182 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.907155    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:07:50.947159 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.947137    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:07:50.950828 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.950805    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e74ae40-4e63-49f0-a734-d41735891c52" containerID="eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5" exitCode=0
Sep 16 01:07:50.950962 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.950925    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46" event={"ID":"2e74ae40-4e63-49f0-a734-d41735891c52","Type":"ContainerDied","Data":"eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5"}
Sep 16 01:07:50.950962 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.950924    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"
Sep 16 01:07:50.950962 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.950957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46" event={"ID":"2e74ae40-4e63-49f0-a734-d41735891c52","Type":"ContainerDied","Data":"9445b9e9cbd8ab1823e37bfa35650b558111075cf7c7a7da0eaa998af20bc542"}
Sep 16 01:07:50.951126 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.950980    2573 scope.go:117] "RemoveContainer" containerID="eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5"
Sep 16 01:07:50.952426 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.952402    2573 generic.go:356] "Generic (PLEG): container finished" podID="00dd3601-90c7-4ac2-94a5-115d870b78ee" containerID="d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5" exitCode=0
Sep 16 01:07:50.952616 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.952502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh" event={"ID":"00dd3601-90c7-4ac2-94a5-115d870b78ee","Type":"ContainerDied","Data":"d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5"}
Sep 16 01:07:50.952616 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.952554    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh" event={"ID":"00dd3601-90c7-4ac2-94a5-115d870b78ee","Type":"ContainerDied","Data":"07e659105bcec2335af285d38e9b1423047d785136f5e6a350bf57d076647abf"}
Sep 16 01:07:50.952616 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.952509    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"
Sep 16 01:07:50.962624 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.962490    2573 scope.go:117] "RemoveContainer" containerID="eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5"
Sep 16 01:07:50.962874 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:07:50.962846    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5\": container with ID starting with eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5 not found: ID does not exist" containerID="eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5"
Sep 16 01:07:50.962948 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.962886    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5"} err="failed to get container status \"eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5\": rpc error: code = NotFound desc = could not find container \"eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5\": container with ID starting with eaec1951b383dc97d97020a11b9a480e9abb4a4edef36607f7435837d67dfaa5 not found: ID does not exist"
Sep 16 01:07:50.962948 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.962910    2573 scope.go:117] "RemoveContainer" containerID="d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5"
Sep 16 01:07:50.976037 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.976009    2573 scope.go:117] "RemoveContainer" containerID="d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5"
Sep 16 01:07:50.976417 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:07:50.976384    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5\": container with ID starting with d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5 not found: ID does not exist" containerID="d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5"
Sep 16 01:07:50.976503 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:50.976424    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5"} err="failed to get container status \"d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5\": rpc error: code = NotFound desc = could not find container \"d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5\": container with ID starting with d49d678e8837c4ab0deab23ec3743ca1c2e25ceafae74e1c2b6ec222b1a894a5 not found: ID does not exist"
Sep 16 01:07:51.060603 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060563    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-home\") pod \"2e74ae40-4e63-49f0-a734-d41735891c52\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") "
Sep 16 01:07:51.060779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060620    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-dshm\") pod \"00dd3601-90c7-4ac2-94a5-115d870b78ee\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") "
Sep 16 01:07:51.060779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060656    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-model-cache\") pod \"2e74ae40-4e63-49f0-a734-d41735891c52\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") "
Sep 16 01:07:51.060779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060731    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-tmp-dir\") pod \"00dd3601-90c7-4ac2-94a5-115d870b78ee\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") "
Sep 16 01:07:51.060779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060757    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-home\") pod \"00dd3601-90c7-4ac2-94a5-115d870b78ee\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") "
Sep 16 01:07:51.060968 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060806    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hngf2\" (UniqueName: \"kubernetes.io/projected/2e74ae40-4e63-49f0-a734-d41735891c52-kube-api-access-hngf2\") pod \"2e74ae40-4e63-49f0-a734-d41735891c52\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") "
Sep 16 01:07:51.060968 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060832    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-home" (OuterVolumeSpecName: "home") pod "2e74ae40-4e63-49f0-a734-d41735891c52" (UID: "2e74ae40-4e63-49f0-a734-d41735891c52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:51.060968 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060862    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fcpr\" (UniqueName: \"kubernetes.io/projected/00dd3601-90c7-4ac2-94a5-115d870b78ee-kube-api-access-7fcpr\") pod \"00dd3601-90c7-4ac2-94a5-115d870b78ee\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") "
Sep 16 01:07:51.060968 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.060935    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-tmp-dir\") pod \"2e74ae40-4e63-49f0-a734-d41735891c52\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") "
Sep 16 01:07:51.061172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061008    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e74ae40-4e63-49f0-a734-d41735891c52-tls-certs\") pod \"2e74ae40-4e63-49f0-a734-d41735891c52\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") "
Sep 16 01:07:51.061172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061011    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00dd3601-90c7-4ac2-94a5-115d870b78ee" (UID: "00dd3601-90c7-4ac2-94a5-115d870b78ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:51.061172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061011    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e74ae40-4e63-49f0-a734-d41735891c52" (UID: "2e74ae40-4e63-49f0-a734-d41735891c52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:51.061172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061066    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-home" (OuterVolumeSpecName: "home") pod "00dd3601-90c7-4ac2-94a5-115d870b78ee" (UID: "00dd3601-90c7-4ac2-94a5-115d870b78ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:51.061172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061079    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-model-cache\") pod \"00dd3601-90c7-4ac2-94a5-115d870b78ee\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") "
Sep 16 01:07:51.061172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061106    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-dshm\") pod \"2e74ae40-4e63-49f0-a734-d41735891c52\" (UID: \"2e74ae40-4e63-49f0-a734-d41735891c52\") "
Sep 16 01:07:51.061172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061151    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00dd3601-90c7-4ac2-94a5-115d870b78ee-tls-certs\") pod \"00dd3601-90c7-4ac2-94a5-115d870b78ee\" (UID: \"00dd3601-90c7-4ac2-94a5-115d870b78ee\") "
Sep 16 01:07:51.061608 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061248    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e74ae40-4e63-49f0-a734-d41735891c52" (UID: "2e74ae40-4e63-49f0-a734-d41735891c52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:51.061608 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061291    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "00dd3601-90c7-4ac2-94a5-115d870b78ee" (UID: "00dd3601-90c7-4ac2-94a5-115d870b78ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:51.061849 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061821    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.061849 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061843    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.061956 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061856    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.061956 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061868    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.061956 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061880    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.061956 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.061891    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.063482 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.063456    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e74ae40-4e63-49f0-a734-d41735891c52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e74ae40-4e63-49f0-a734-d41735891c52" (UID: "2e74ae40-4e63-49f0-a734-d41735891c52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:07:51.063482 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.063468    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dd3601-90c7-4ac2-94a5-115d870b78ee-kube-api-access-7fcpr" (OuterVolumeSpecName: "kube-api-access-7fcpr") pod "00dd3601-90c7-4ac2-94a5-115d870b78ee" (UID: "00dd3601-90c7-4ac2-94a5-115d870b78ee"). InnerVolumeSpecName "kube-api-access-7fcpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:51.063695 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.063672    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-dshm" (OuterVolumeSpecName: "dshm") pod "00dd3601-90c7-4ac2-94a5-115d870b78ee" (UID: "00dd3601-90c7-4ac2-94a5-115d870b78ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:51.063853 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.063838    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-dshm" (OuterVolumeSpecName: "dshm") pod "2e74ae40-4e63-49f0-a734-d41735891c52" (UID: "2e74ae40-4e63-49f0-a734-d41735891c52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:51.063919 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.063903    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e74ae40-4e63-49f0-a734-d41735891c52-kube-api-access-hngf2" (OuterVolumeSpecName: "kube-api-access-hngf2") pod "2e74ae40-4e63-49f0-a734-d41735891c52" (UID: "2e74ae40-4e63-49f0-a734-d41735891c52"). InnerVolumeSpecName "kube-api-access-hngf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:51.063972 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.063927    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00dd3601-90c7-4ac2-94a5-115d870b78ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00dd3601-90c7-4ac2-94a5-115d870b78ee" (UID: "00dd3601-90c7-4ac2-94a5-115d870b78ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:07:51.162829 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.162755    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hngf2\" (UniqueName: \"kubernetes.io/projected/2e74ae40-4e63-49f0-a734-d41735891c52-kube-api-access-hngf2\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.162829 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.162780    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fcpr\" (UniqueName: \"kubernetes.io/projected/00dd3601-90c7-4ac2-94a5-115d870b78ee-kube-api-access-7fcpr\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.162829 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.162789    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e74ae40-4e63-49f0-a734-d41735891c52-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.162829 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.162799    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e74ae40-4e63-49f0-a734-d41735891c52-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.162829 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.162807    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00dd3601-90c7-4ac2-94a5-115d870b78ee-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.162829 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.162814    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00dd3601-90c7-4ac2-94a5-115d870b78ee-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:51.274713 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.274687    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"]
Sep 16 01:07:51.281123 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.281100    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/simulator-w3-2534e3-kserve-5cf84f7c86-lhqvh"]
Sep 16 01:07:51.290863 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.290833    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"]
Sep 16 01:07:51.294514 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:51.294494    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2534e3/distinct-w3-2534e3-kserve-6944dd8998-64b46"]
Sep 16 01:07:52.771715 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:52.771684    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00dd3601-90c7-4ac2-94a5-115d870b78ee" path="/var/lib/kubelet/pods/00dd3601-90c7-4ac2-94a5-115d870b78ee/volumes"
Sep 16 01:07:52.772169 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:52.772115    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e74ae40-4e63-49f0-a734-d41735891c52" path="/var/lib/kubelet/pods/2e74ae40-4e63-49f0-a734-d41735891c52/volumes"
Sep 16 01:07:53.019375 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:53.019340    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"]
Sep 16 01:07:53.019633 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:53.019603    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" containerID="cri-o://c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d" gracePeriod=30
Sep 16 01:07:56.521882 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:56.521845    2573 patch_prober.go:28] interesting pod/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 01:07:56.522241 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:56.521919    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 01:07:57.765595 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:57.765558    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:07:58.281912 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.281889    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:07:58.337777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.337697    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-socket\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.337777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.337748    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87589a4b-1114-4d80-91b3-34f5270b9b6d-istiod-ca-cert\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.337965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.337794    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-podinfo\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.337965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.337827    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxzhk\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-kube-api-access-bxzhk\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.337965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.337865    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-token\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.337965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.337922    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-envoy\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.337965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.337957    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-data\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.338226 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.337977    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:58.338226 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.338004    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-certs\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.338226 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.338057    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-credential-socket\") pod \"87589a4b-1114-4d80-91b3-34f5270b9b6d\" (UID: \"87589a4b-1114-4d80-91b3-34f5270b9b6d\") "
Sep 16 01:07:58.338226 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.338184    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87589a4b-1114-4d80-91b3-34f5270b9b6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:07:58.338434 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.338416    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:58.338493 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.338446    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:58.338493 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.338469    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87589a4b-1114-4d80-91b3-34f5270b9b6d-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:58.338648 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.338617    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:58.338775 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.338713    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:58.340608 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.340404    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-kube-api-access-bxzhk" (OuterVolumeSpecName: "kube-api-access-bxzhk") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "kube-api-access-bxzhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:58.340716 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.340608    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:07:58.340716 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.340702    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:07:58.340839 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.340752    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87589a4b-1114-4d80-91b3-34f5270b9b6d" (UID: "87589a4b-1114-4d80-91b3-34f5270b9b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:07:58.439259 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.439227    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:58.439259 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.439254    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxzhk\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-kube-api-access-bxzhk\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:58.439259 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.439264    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:58.439441 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.439272    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:58.439441 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.439280    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:58.439441 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.439288    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:58.439441 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:58.439297    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87589a4b-1114-4d80-91b3-34f5270b9b6d-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:07:59.002937 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.002907    2573 generic.go:356] "Generic (PLEG): container finished" podID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerID="c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d" exitCode=0
Sep 16 01:07:59.003356 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.002966    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"
Sep 16 01:07:59.003356 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.002995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" event={"ID":"87589a4b-1114-4d80-91b3-34f5270b9b6d","Type":"ContainerDied","Data":"c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d"}
Sep 16 01:07:59.003356 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.003044    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj" event={"ID":"87589a4b-1114-4d80-91b3-34f5270b9b6d","Type":"ContainerDied","Data":"c7e5181cbfe2c7f26aaf4410f71e7d132bd057f7045d51be792284dba0a052c2"}
Sep 16 01:07:59.003356 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.003066    2573 scope.go:117] "RemoveContainer" containerID="c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d"
Sep 16 01:07:59.012652 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.012638    2573 scope.go:117] "RemoveContainer" containerID="6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504"
Sep 16 01:07:59.020989 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.020974    2573 scope.go:117] "RemoveContainer" containerID="c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d"
Sep 16 01:07:59.021182 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.021159    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"]
Sep 16 01:07:59.021269 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:07:59.021252    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d\": container with ID starting with c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d not found: ID does not exist" containerID="c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d"
Sep 16 01:07:59.021310 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.021277    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d"} err="failed to get container status \"c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d\": rpc error: code = NotFound desc = could not find container \"c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d\": container with ID starting with c1e759bec85c41712af1476e62c7ffa0344853ce5edfa28d824beb0e4cc26e0d not found: ID does not exist"
Sep 16 01:07:59.021310 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.021291    2573 scope.go:117] "RemoveContainer" containerID="6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504"
Sep 16 01:07:59.021586 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:07:59.021567    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504\": container with ID starting with 6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504 not found: ID does not exist" containerID="6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504"
Sep 16 01:07:59.021652 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.021588    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504"} err="failed to get container status \"6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504\": rpc error: code = NotFound desc = could not find container \"6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504\": container with ID starting with 6177f2476268dea479e20c82507a23006df3c658cf1762aed255957ff443a504 not found: ID does not exist"
Sep 16 01:07:59.026196 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:07:59.026176    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2534e3-openshift-default-79f6c745d6-6pnpj"]
Sep 16 01:08:00.771618 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:00.771581    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" path="/var/lib/kubelet/pods/87589a4b-1114-4d80-91b3-34f5270b9b6d/volumes"
Sep 16 01:08:25.765171 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:25.765128    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:08:34.877332 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:34.877301    2573 scope.go:117] "RemoveContainer" containerID="a85c6542551d9bd852d010bc78644103c54d677e75ac03d43738c989aff05362"
Sep 16 01:08:41.684300 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684267    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"]
Sep 16 01:08:41.684713 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684700    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684715    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684731    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e74ae40-4e63-49f0-a734-d41735891c52" containerName="main"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684737    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e74ae40-4e63-49f0-a734-d41735891c52" containerName="main"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684748    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00dd3601-90c7-4ac2-94a5-115d870b78ee" containerName="main"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684755    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="00dd3601-90c7-4ac2-94a5-115d870b78ee" containerName="main"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684760    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ed49c0-93dd-4a19-95b4-9a53754e32ab" containerName="revoke-keys"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684766    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ed49c0-93dd-4a19-95b4-9a53754e32ab" containerName="revoke-keys"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684776    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26927ad2-6d42-4ab7-bde1-6d363b14afcd" containerName="payload-processing"
Sep 16 01:08:41.684783 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684782    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="26927ad2-6d42-4ab7-bde1-6d363b14afcd" containerName="payload-processing"
Sep 16 01:08:41.685195 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684845    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="00dd3601-90c7-4ac2-94a5-115d870b78ee" containerName="main"
Sep 16 01:08:41.685195 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684854    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy"
Sep 16 01:08:41.685195 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684860    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e74ae40-4e63-49f0-a734-d41735891c52" containerName="main"
Sep 16 01:08:41.685195 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684867    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ed49c0-93dd-4a19-95b4-9a53754e32ab" containerName="revoke-keys"
Sep 16 01:08:41.685195 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.684873    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="26927ad2-6d42-4ab7-bde1-6d363b14afcd" containerName="payload-processing"
Sep 16 01:08:41.689643 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.689627    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.691603 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.691586    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-96c840\"/\"embedding-w0-96c840-kserve-self-signed-certs\""
Sep 16 01:08:41.696603 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.696579    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"]
Sep 16 01:08:41.736260 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.736232    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tls-certs\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.736397 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.736269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-model-cache\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.736397 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.736325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-home\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.736397 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.736349    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-dshm\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.736506 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.736395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ngl\" (UniqueName: \"kubernetes.io/projected/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-kube-api-access-s8ngl\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.736506 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.736465    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tmp-dir\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.838796 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.838483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tmp-dir\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.838796 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.838735    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tls-certs\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.839075 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.838863    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tmp-dir\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.839075 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.838930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-model-cache\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.839075 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.839047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-home\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.839247 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.839114    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-dshm\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.839247 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.839158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ngl\" (UniqueName: \"kubernetes.io/projected/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-kube-api-access-s8ngl\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.839247 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.839231    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-model-cache\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.839486 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.839462    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-home\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.842307 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.842282    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tls-certs\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.842406 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.842382    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-dshm\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:41.846740 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:41.846713    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ngl\" (UniqueName: \"kubernetes.io/projected/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-kube-api-access-s8ngl\") pod \"embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") " pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:42.000223 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:42.000190    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:42.274014 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:42.273987    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"]
Sep 16 01:08:42.276037 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:08:42.276011    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeee95b02_5eb5_4bd0_bdc2_6c7610dfb375.slice/crio-e02d8315f022d3867a9f776985b2bc490adfdd1662367c8f630f99c0d4683e67 WatchSource:0}: Error finding container e02d8315f022d3867a9f776985b2bc490adfdd1662367c8f630f99c0d4683e67: Status 404 returned error can't find the container with id e02d8315f022d3867a9f776985b2bc490adfdd1662367c8f630f99c0d4683e67
Sep 16 01:08:43.267693 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:43.267655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2" event={"ID":"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375","Type":"ContainerStarted","Data":"2bd10507884e4fa4878d7b38eba23c4660fe6578bb40f1992e7d942e24cfbd18"}
Sep 16 01:08:43.268074 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:43.267701    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2" event={"ID":"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375","Type":"ContainerStarted","Data":"e02d8315f022d3867a9f776985b2bc490adfdd1662367c8f630f99c0d4683e67"}
Sep 16 01:08:43.284907 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:43.284857    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2" podStartSLOduration=2.284843936 podStartE2EDuration="2.284843936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:08:43.281727494 +0000 UTC m=+1508.984549700" watchObservedRunningTime="2026-09-16 01:08:43.284843936 +0000 UTC m=+1508.987666131"
Sep 16 01:08:52.000298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.000268    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:52.000690 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.000312    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:52.013070 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.013042    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:52.181332 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.181298    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"]
Sep 16 01:08:52.182012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.181987    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy"
Sep 16 01:08:52.182012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.182011    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy"
Sep 16 01:08:52.182232 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.182124    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87589a4b-1114-4d80-91b3-34f5270b9b6d" containerName="istio-proxy"
Sep 16 01:08:52.185741 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.185722    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.188105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.187908    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"default-dockercfg-rnjz5\""
Sep 16 01:08:52.188105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.187963    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"openshift-service-ca.crt\""
Sep 16 01:08:52.188105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.187990    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"auth-test-model-0ee1bf-kserve-self-signed-certs\""
Sep 16 01:08:52.188105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.187962    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"kube-root-ca.crt\""
Sep 16 01:08:52.195281 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.195236    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"]
Sep 16 01:08:52.237485 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.237460    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrkq\" (UniqueName: \"kubernetes.io/projected/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-kube-api-access-hdrkq\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.237631 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.237507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-model-cache\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.237685 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.237638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-dshm\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.237730 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.237682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tmp-dir\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.237769 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.237740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tls-certs\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.237810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.237794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-home\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.334369 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.334290    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:08:52.339030 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.338965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tls-certs\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.339030 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.339023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-home\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.339215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.339071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrkq\" (UniqueName: \"kubernetes.io/projected/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-kube-api-access-hdrkq\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.339298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.339272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-model-cache\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.339403 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.339386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-dshm\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.339484 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.339440    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tmp-dir\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.339684 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.339666    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-home\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.339684 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.339683    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-model-cache\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.339823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.339737    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tmp-dir\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.341692 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.341668    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-dshm\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.341894 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.341874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tls-certs\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.346634 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.346608    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrkq\" (UniqueName: \"kubernetes.io/projected/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-kube-api-access-hdrkq\") pod \"auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.499235 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.499192    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:08:52.775762 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:52.775739    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"]
Sep 16 01:08:52.778455 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:08:52.778424    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ceb9338_dcf1_4c85_885d_16d3f2c546cb.slice/crio-87008a3123f36f61839af70dcaa2969d547f1f26011b67f66334ecfc38a8430f WatchSource:0}: Error finding container 87008a3123f36f61839af70dcaa2969d547f1f26011b67f66334ecfc38a8430f: Status 404 returned error can't find the container with id 87008a3123f36f61839af70dcaa2969d547f1f26011b67f66334ecfc38a8430f
Sep 16 01:08:53.331210 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:53.331170    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk" event={"ID":"4ceb9338-dcf1-4c85-885d-16d3f2c546cb","Type":"ContainerStarted","Data":"f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535"}
Sep 16 01:08:53.331732 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:53.331227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk" event={"ID":"4ceb9338-dcf1-4c85-885d-16d3f2c546cb","Type":"ContainerStarted","Data":"87008a3123f36f61839af70dcaa2969d547f1f26011b67f66334ecfc38a8430f"}
Sep 16 01:08:53.349675 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:08:53.349623    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk" podStartSLOduration=1.3496010250000001 podStartE2EDuration="1.349601025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:08:53.348405207 +0000 UTC m=+1519.051227398" watchObservedRunningTime="2026-09-16 01:08:53.349601025 +0000 UTC m=+1519.052423217"
Sep 16 01:09:02.499799 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:02.499721    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:09:02.499799 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:02.499774    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:09:02.512627 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:02.512604    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:09:03.399665 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:03.399634    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:09:07.765151 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:07.765119    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:09:12.629769 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.629736    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"]
Sep 16 01:09:12.634183 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.634156    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:12.636396 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.636371    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f29ccae3\""
Sep 16 01:09:12.636542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.636412    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f29ccae3-dockercfg-ncgcq\""
Sep 16 01:09:12.643936 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.643913    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"]
Sep 16 01:09:12.747132 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.747098    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e5d02e-293f-49a4-8b63-038e112003f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r\" (UID: \"b9e5d02e-293f-49a4-8b63-038e112003f2\") " pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:12.747323 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.747186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkx4\" (UniqueName: \"kubernetes.io/projected/b9e5d02e-293f-49a4-8b63-038e112003f2-kube-api-access-wlkx4\") pod \"payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r\" (UID: \"b9e5d02e-293f-49a4-8b63-038e112003f2\") " pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:12.848459 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.848425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e5d02e-293f-49a4-8b63-038e112003f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r\" (UID: \"b9e5d02e-293f-49a4-8b63-038e112003f2\") " pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:12.848661 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.848486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkx4\" (UniqueName: \"kubernetes.io/projected/b9e5d02e-293f-49a4-8b63-038e112003f2-kube-api-access-wlkx4\") pod \"payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r\" (UID: \"b9e5d02e-293f-49a4-8b63-038e112003f2\") " pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:12.849208 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.849183    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e5d02e-293f-49a4-8b63-038e112003f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r\" (UID: \"b9e5d02e-293f-49a4-8b63-038e112003f2\") " pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:12.855592 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.855572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkx4\" (UniqueName: \"kubernetes.io/projected/b9e5d02e-293f-49a4-8b63-038e112003f2-kube-api-access-wlkx4\") pod \"payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r\" (UID: \"b9e5d02e-293f-49a4-8b63-038e112003f2\") " pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:12.947541 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:12.947512    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:13.224924 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:13.224892    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"]
Sep 16 01:09:13.226720 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:09:13.226688    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e5d02e_293f_49a4_8b63_038e112003f2.slice/crio-9a8eef8386831a0a4678684b7970dfdf4f9e62db5ffb6ea144b6eb3ad840121e WatchSource:0}: Error finding container 9a8eef8386831a0a4678684b7970dfdf4f9e62db5ffb6ea144b6eb3ad840121e: Status 404 returned error can't find the container with id 9a8eef8386831a0a4678684b7970dfdf4f9e62db5ffb6ea144b6eb3ad840121e
Sep 16 01:09:13.445131 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:13.445089    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r" event={"ID":"b9e5d02e-293f-49a4-8b63-038e112003f2","Type":"ContainerStarted","Data":"520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485"}
Sep 16 01:09:13.445309 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:13.445149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r" event={"ID":"b9e5d02e-293f-49a4-8b63-038e112003f2","Type":"ContainerStarted","Data":"9a8eef8386831a0a4678684b7970dfdf4f9e62db5ffb6ea144b6eb3ad840121e"}
Sep 16 01:09:13.445309 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:13.445172    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:13.462954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:13.462901    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r" podStartSLOduration=1.462885492 podStartE2EDuration="1.462885492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:09:13.460542393 +0000 UTC m=+1539.163364587" watchObservedRunningTime="2026-09-16 01:09:13.462885492 +0000 UTC m=+1539.165707687"
Sep 16 01:09:24.453563 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:24.453535    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:37.620443 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.617251    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"]
Sep 16 01:09:37.623947 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.623920    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"
Sep 16 01:09:37.625932 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.625905    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"]
Sep 16 01:09:37.626286 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.626268    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:09:37.626371 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.626272    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:09:37.684446 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.684417    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d567b866-d583-4867-9b40-405211184f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd\" (UID: \"d567b866-d583-4867-9b40-405211184f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"
Sep 16 01:09:37.785422 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.785392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d567b866-d583-4867-9b40-405211184f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd\" (UID: \"d567b866-d583-4867-9b40-405211184f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"
Sep 16 01:09:37.785995 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.785975    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d567b866-d583-4867-9b40-405211184f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd\" (UID: \"d567b866-d583-4867-9b40-405211184f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"
Sep 16 01:09:37.937878 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:37.937802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"
Sep 16 01:09:38.210563 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:38.210535    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"]
Sep 16 01:09:38.212375 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:09:38.212343    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd567b866_d583_4867_9b40_405211184f68.slice/crio-73bdeff1d8b1f679162588ce681e82d64c3b3c5b3b8a18a535cd9632067f9d45 WatchSource:0}: Error finding container 73bdeff1d8b1f679162588ce681e82d64c3b3c5b3b8a18a535cd9632067f9d45: Status 404 returned error can't find the container with id 73bdeff1d8b1f679162588ce681e82d64c3b3c5b3b8a18a535cd9632067f9d45
Sep 16 01:09:38.605261 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:38.605231    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd" event={"ID":"d567b866-d583-4867-9b40-405211184f68","Type":"ContainerStarted","Data":"8314db765cfd1f2223c23484a589eda836c44c941792a70ec57813fdc5214495"}
Sep 16 01:09:38.605416 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:38.605283    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd" event={"ID":"d567b866-d583-4867-9b40-405211184f68","Type":"ContainerStarted","Data":"73bdeff1d8b1f679162588ce681e82d64c3b3c5b3b8a18a535cd9632067f9d45"}
Sep 16 01:09:38.627546 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:38.627489    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd" podStartSLOduration=1.627472997 podStartE2EDuration="1.627472997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:09:38.624967971 +0000 UTC m=+1564.327790166" watchObservedRunningTime="2026-09-16 01:09:38.627472997 +0000 UTC m=+1564.330295220"
Sep 16 01:09:39.613717 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:39.613676    2573 generic.go:356] "Generic (PLEG): container finished" podID="d567b866-d583-4867-9b40-405211184f68" containerID="8314db765cfd1f2223c23484a589eda836c44c941792a70ec57813fdc5214495" exitCode=0
Sep 16 01:09:39.613897 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:39.613761    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd" event={"ID":"d567b866-d583-4867-9b40-405211184f68","Type":"ContainerDied","Data":"8314db765cfd1f2223c23484a589eda836c44c941792a70ec57813fdc5214495"}
Sep 16 01:09:40.788193 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:40.788167    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"
Sep 16 01:09:40.923476 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:40.923398    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d567b866-d583-4867-9b40-405211184f68-maas-api-service-ca\") pod \"d567b866-d583-4867-9b40-405211184f68\" (UID: \"d567b866-d583-4867-9b40-405211184f68\") "
Sep 16 01:09:40.923721 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:40.923700    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d567b866-d583-4867-9b40-405211184f68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d567b866-d583-4867-9b40-405211184f68" (UID: "d567b866-d583-4867-9b40-405211184f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:09:40.923837 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:40.923824    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d567b866-d583-4867-9b40-405211184f68-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:41.628350 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:41.628317    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"
Sep 16 01:09:41.628350 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:41.628336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd" event={"ID":"d567b866-d583-4867-9b40-405211184f68","Type":"ContainerDied","Data":"73bdeff1d8b1f679162588ce681e82d64c3b3c5b3b8a18a535cd9632067f9d45"}
Sep 16 01:09:41.628609 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:41.628387    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73bdeff1d8b1f679162588ce681e82d64c3b3c5b3b8a18a535cd9632067f9d45"
Sep 16 01:09:42.484105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.484071    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"]
Sep 16 01:09:42.497143 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.497111    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"]
Sep 16 01:09:42.545007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.544976    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"]
Sep 16 01:09:42.545452 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.545440    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d567b866-d583-4867-9b40-405211184f68" containerName="revoke-keys"
Sep 16 01:09:42.545495 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.545454    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d567b866-d583-4867-9b40-405211184f68" containerName="revoke-keys"
Sep 16 01:09:42.545556 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.545536    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d567b866-d583-4867-9b40-405211184f68" containerName="revoke-keys"
Sep 16 01:09:42.548872 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.548850    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"
Sep 16 01:09:42.550879 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.550860    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:09:42.550979 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.550903    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:09:42.555205 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.555181    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"]
Sep 16 01:09:42.649484 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.649451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902400d2-31de-421c-8d5a-04b9c284c078-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn\" (UID: \"902400d2-31de-421c-8d5a-04b9c284c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"
Sep 16 01:09:42.750871 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.750787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902400d2-31de-421c-8d5a-04b9c284c078-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn\" (UID: \"902400d2-31de-421c-8d5a-04b9c284c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"
Sep 16 01:09:42.751361 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.751343    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902400d2-31de-421c-8d5a-04b9c284c078-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn\" (UID: \"902400d2-31de-421c-8d5a-04b9c284c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"
Sep 16 01:09:42.860932 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:42.860898    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"
Sep 16 01:09:43.136661 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:43.136583    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"]
Sep 16 01:09:43.140227 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:09:43.140200    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902400d2_31de_421c_8d5a_04b9c284c078.slice/crio-159e566f29590556293cb8902b8407d431bb1db6437b873096e149e3018a3f7a WatchSource:0}: Error finding container 159e566f29590556293cb8902b8407d431bb1db6437b873096e149e3018a3f7a: Status 404 returned error can't find the container with id 159e566f29590556293cb8902b8407d431bb1db6437b873096e149e3018a3f7a
Sep 16 01:09:43.645343 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:43.645304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn" event={"ID":"902400d2-31de-421c-8d5a-04b9c284c078","Type":"ContainerStarted","Data":"87b52bca813a2881dc621c4ee0060dd3c78bec4d6971c12cdee080735cd1dbb7"}
Sep 16 01:09:43.645727 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:43.645364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn" event={"ID":"902400d2-31de-421c-8d5a-04b9c284c078","Type":"ContainerStarted","Data":"159e566f29590556293cb8902b8407d431bb1db6437b873096e149e3018a3f7a"}
Sep 16 01:09:43.660282 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:43.660233    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn" podStartSLOduration=1.6602180149999999 podStartE2EDuration="1.660218015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:09:43.658339256 +0000 UTC m=+1569.361161452" watchObservedRunningTime="2026-09-16 01:09:43.660218015 +0000 UTC m=+1569.363040238"
Sep 16 01:09:44.654153 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:44.654023    2573 generic.go:356] "Generic (PLEG): container finished" podID="902400d2-31de-421c-8d5a-04b9c284c078" containerID="87b52bca813a2881dc621c4ee0060dd3c78bec4d6971c12cdee080735cd1dbb7" exitCode=0
Sep 16 01:09:44.654153 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:44.654067    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn" event={"ID":"902400d2-31de-421c-8d5a-04b9c284c078","Type":"ContainerDied","Data":"87b52bca813a2881dc621c4ee0060dd3c78bec4d6971c12cdee080735cd1dbb7"}
Sep 16 01:09:45.800575 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:45.800544    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"
Sep 16 01:09:45.981976 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:45.981944    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902400d2-31de-421c-8d5a-04b9c284c078-maas-api-service-ca\") pod \"902400d2-31de-421c-8d5a-04b9c284c078\" (UID: \"902400d2-31de-421c-8d5a-04b9c284c078\") "
Sep 16 01:09:45.982313 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:45.982294    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902400d2-31de-421c-8d5a-04b9c284c078-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "902400d2-31de-421c-8d5a-04b9c284c078" (UID: "902400d2-31de-421c-8d5a-04b9c284c078"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:09:46.083631 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:46.083604    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902400d2-31de-421c-8d5a-04b9c284c078-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:46.668487 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:46.668457    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"
Sep 16 01:09:46.668678 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:46.668456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn" event={"ID":"902400d2-31de-421c-8d5a-04b9c284c078","Type":"ContainerDied","Data":"159e566f29590556293cb8902b8407d431bb1db6437b873096e149e3018a3f7a"}
Sep 16 01:09:46.668678 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:46.668589    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159e566f29590556293cb8902b8407d431bb1db6437b873096e149e3018a3f7a"
Sep 16 01:09:49.764679 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:49.764644    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:09:53.278238 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.278197    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"]
Sep 16 01:09:53.278770 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.278424    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r" podUID="b9e5d02e-293f-49a4-8b63-038e112003f2" containerName="payload-processing" containerID="cri-o://520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485" gracePeriod=30
Sep 16 01:09:53.454007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.453978    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"]
Sep 16 01:09:53.457092 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.457068    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f29ccae3-9cdbn"]
Sep 16 01:09:53.566102 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.566083    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:53.659344 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.657906    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlkx4\" (UniqueName: \"kubernetes.io/projected/b9e5d02e-293f-49a4-8b63-038e112003f2-kube-api-access-wlkx4\") pod \"b9e5d02e-293f-49a4-8b63-038e112003f2\" (UID: \"b9e5d02e-293f-49a4-8b63-038e112003f2\") "
Sep 16 01:09:53.659344 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.657972    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e5d02e-293f-49a4-8b63-038e112003f2-plugins-config-volume\") pod \"b9e5d02e-293f-49a4-8b63-038e112003f2\" (UID: \"b9e5d02e-293f-49a4-8b63-038e112003f2\") "
Sep 16 01:09:53.659344 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.658767    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e5d02e-293f-49a4-8b63-038e112003f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9e5d02e-293f-49a4-8b63-038e112003f2" (UID: "b9e5d02e-293f-49a4-8b63-038e112003f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:09:53.676008 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.675943    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e5d02e-293f-49a4-8b63-038e112003f2-kube-api-access-wlkx4" (OuterVolumeSpecName: "kube-api-access-wlkx4") pod "b9e5d02e-293f-49a4-8b63-038e112003f2" (UID: "b9e5d02e-293f-49a4-8b63-038e112003f2"). InnerVolumeSpecName "kube-api-access-wlkx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:09:53.713458 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.713421    2573 generic.go:356] "Generic (PLEG): container finished" podID="b9e5d02e-293f-49a4-8b63-038e112003f2" containerID="520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485" exitCode=0
Sep 16 01:09:53.713458 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.713459    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"
Sep 16 01:09:53.713687 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.713482    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r" event={"ID":"b9e5d02e-293f-49a4-8b63-038e112003f2","Type":"ContainerDied","Data":"520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485"}
Sep 16 01:09:53.713687 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.713548    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r" event={"ID":"b9e5d02e-293f-49a4-8b63-038e112003f2","Type":"ContainerDied","Data":"9a8eef8386831a0a4678684b7970dfdf4f9e62db5ffb6ea144b6eb3ad840121e"}
Sep 16 01:09:53.713687 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.713573    2573 scope.go:117] "RemoveContainer" containerID="520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485"
Sep 16 01:09:53.723863 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.723835    2573 scope.go:117] "RemoveContainer" containerID="520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485"
Sep 16 01:09:53.724112 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:09:53.724095    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485\": container with ID starting with 520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485 not found: ID does not exist" containerID="520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485"
Sep 16 01:09:53.724178 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.724120    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485"} err="failed to get container status \"520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485\": rpc error: code = NotFound desc = could not find container \"520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485\": container with ID starting with 520ae90d4e57784714643cc1bf07e10d683731ff92579711d7ad99d32bcb4485 not found: ID does not exist"
Sep 16 01:09:53.735378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.735350    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"]
Sep 16 01:09:53.739142 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.739121    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f29ccae3-c47b8bc49-b4w4r"]
Sep 16 01:09:53.759145 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.759115    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlkx4\" (UniqueName: \"kubernetes.io/projected/b9e5d02e-293f-49a4-8b63-038e112003f2-kube-api-access-wlkx4\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:53.759145 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.759136    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e5d02e-293f-49a4-8b63-038e112003f2-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:53.963173 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.963135    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"]
Sep 16 01:09:53.966353 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:53.966328    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-96c840-smxdd"]
Sep 16 01:09:54.771111 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:54.771079    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902400d2-31de-421c-8d5a-04b9c284c078" path="/var/lib/kubelet/pods/902400d2-31de-421c-8d5a-04b9c284c078/volumes"
Sep 16 01:09:54.771483 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:54.771431    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e5d02e-293f-49a4-8b63-038e112003f2" path="/var/lib/kubelet/pods/b9e5d02e-293f-49a4-8b63-038e112003f2/volumes"
Sep 16 01:09:54.771755 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:54.771741    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d567b866-d583-4867-9b40-405211184f68" path="/var/lib/kubelet/pods/d567b866-d583-4867-9b40-405211184f68/volumes"
Sep 16 01:09:57.505141 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.505072    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q" podUID="4976c5c6-a606-43ec-b334-edd7b718a053" containerName="main" containerID="cri-o://2525eba794e15b2f491d36d69ece18febccb3c432371a7cf274b1c93dc2ef5cb" gracePeriod=45
Sep 16 01:09:57.518211 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.518172    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2" podUID="eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" containerName="main" containerID="cri-o://2bd10507884e4fa4878d7b38eba23c4660fe6578bb40f1992e7d942e24cfbd18" gracePeriod=45
Sep 16 01:09:57.748000 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.747956    2573 generic.go:356] "Generic (PLEG): container finished" podID="4976c5c6-a606-43ec-b334-edd7b718a053" containerID="2525eba794e15b2f491d36d69ece18febccb3c432371a7cf274b1c93dc2ef5cb" exitCode=0
Sep 16 01:09:57.748163 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.748108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q" event={"ID":"4976c5c6-a606-43ec-b334-edd7b718a053","Type":"ContainerDied","Data":"2525eba794e15b2f491d36d69ece18febccb3c432371a7cf274b1c93dc2ef5cb"}
Sep 16 01:09:57.749908 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.749886    2573 generic.go:356] "Generic (PLEG): container finished" podID="eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" containerID="2bd10507884e4fa4878d7b38eba23c4660fe6578bb40f1992e7d942e24cfbd18" exitCode=0
Sep 16 01:09:57.750036 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.749930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2" event={"ID":"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375","Type":"ContainerDied","Data":"2bd10507884e4fa4878d7b38eba23c4660fe6578bb40f1992e7d942e24cfbd18"}
Sep 16 01:09:57.777469 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.777449    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.800018    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-home\") pod \"4976c5c6-a606-43ec-b334-edd7b718a053\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") "
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.800075    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4976c5c6-a606-43ec-b334-edd7b718a053-tls-certs\") pod \"4976c5c6-a606-43ec-b334-edd7b718a053\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") "
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.800153    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9xhc\" (UniqueName: \"kubernetes.io/projected/4976c5c6-a606-43ec-b334-edd7b718a053-kube-api-access-d9xhc\") pod \"4976c5c6-a606-43ec-b334-edd7b718a053\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") "
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.800234    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-model-cache\") pod \"4976c5c6-a606-43ec-b334-edd7b718a053\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") "
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.800255    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-tmp-dir\") pod \"4976c5c6-a606-43ec-b334-edd7b718a053\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") "
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.800306    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-dshm\") pod \"4976c5c6-a606-43ec-b334-edd7b718a053\" (UID: \"4976c5c6-a606-43ec-b334-edd7b718a053\") "
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.801423    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-model-cache" (OuterVolumeSpecName: "model-cache") pod "4976c5c6-a606-43ec-b334-edd7b718a053" (UID: "4976c5c6-a606-43ec-b334-edd7b718a053"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.801433    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-home" (OuterVolumeSpecName: "home") pod "4976c5c6-a606-43ec-b334-edd7b718a053" (UID: "4976c5c6-a606-43ec-b334-edd7b718a053"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.801651    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4976c5c6-a606-43ec-b334-edd7b718a053" (UID: "4976c5c6-a606-43ec-b334-edd7b718a053"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.803593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.803513    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4976c5c6-a606-43ec-b334-edd7b718a053-kube-api-access-d9xhc" (OuterVolumeSpecName: "kube-api-access-d9xhc") pod "4976c5c6-a606-43ec-b334-edd7b718a053" (UID: "4976c5c6-a606-43ec-b334-edd7b718a053"). InnerVolumeSpecName "kube-api-access-d9xhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:09:57.804246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.803907    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-dshm" (OuterVolumeSpecName: "dshm") pod "4976c5c6-a606-43ec-b334-edd7b718a053" (UID: "4976c5c6-a606-43ec-b334-edd7b718a053"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.804246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.804047    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4976c5c6-a606-43ec-b334-edd7b718a053-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4976c5c6-a606-43ec-b334-edd7b718a053" (UID: "4976c5c6-a606-43ec-b334-edd7b718a053"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:09:57.809132 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.809113    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:09:57.902001 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.901976    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-dshm\") pod \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") "
Sep 16 01:09:57.902194 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902011    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tmp-dir\") pod \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") "
Sep 16 01:09:57.902194 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902039    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-model-cache\") pod \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") "
Sep 16 01:09:57.902194 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902088    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tls-certs\") pod \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") "
Sep 16 01:09:57.902194 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902106    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-home\") pod \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") "
Sep 16 01:09:57.902194 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902133    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8ngl\" (UniqueName: \"kubernetes.io/projected/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-kube-api-access-s8ngl\") pod \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\" (UID: \"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375\") "
Sep 16 01:09:57.902466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902289    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" (UID: "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.902466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902331    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-model-cache" (OuterVolumeSpecName: "model-cache") pod "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" (UID: "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.902466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902376    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-home" (OuterVolumeSpecName: "home") pod "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" (UID: "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902574    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902589    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902600    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902610    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902617    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902624    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4976c5c6-a606-43ec-b334-edd7b718a053-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902631    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902638    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4976c5c6-a606-43ec-b334-edd7b718a053-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.902655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.902647    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xhc\" (UniqueName: \"kubernetes.io/projected/4976c5c6-a606-43ec-b334-edd7b718a053-kube-api-access-d9xhc\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:57.904182 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.904157    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-dshm" (OuterVolumeSpecName: "dshm") pod "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" (UID: "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:09:57.904298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.904281    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" (UID: "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:09:57.904361 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:57.904343    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-kube-api-access-s8ngl" (OuterVolumeSpecName: "kube-api-access-s8ngl") pod "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" (UID: "eee95b02-5eb5-4bd0-bdc2-6c7610dfb375"). InnerVolumeSpecName "kube-api-access-s8ngl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:09:58.003273 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.003248    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:58.003273 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.003274    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8ngl\" (UniqueName: \"kubernetes.io/projected/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-kube-api-access-s8ngl\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:58.003443 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.003284    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:09:58.757299 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.757274    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"
Sep 16 01:09:58.757763 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.757309    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2" event={"ID":"eee95b02-5eb5-4bd0-bdc2-6c7610dfb375","Type":"ContainerDied","Data":"e02d8315f022d3867a9f776985b2bc490adfdd1662367c8f630f99c0d4683e67"}
Sep 16 01:09:58.757763 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.757361    2573 scope.go:117] "RemoveContainer" containerID="2bd10507884e4fa4878d7b38eba23c4660fe6578bb40f1992e7d942e24cfbd18"
Sep 16 01:09:58.759120 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.759095    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q" event={"ID":"4976c5c6-a606-43ec-b334-edd7b718a053","Type":"ContainerDied","Data":"f21e2beb31ed63486ea085fcbf58a1be2eecd8a83e7f387f40ba6dd02eb82e56"}
Sep 16 01:09:58.759120 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.759113    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"
Sep 16 01:09:58.767797 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.767770    2573 scope.go:117] "RemoveContainer" containerID="2525eba794e15b2f491d36d69ece18febccb3c432371a7cf274b1c93dc2ef5cb"
Sep 16 01:09:58.781031 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.781009    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"]
Sep 16 01:09:58.784248 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.784223    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/distinct-2-w0-96c840-kserve-56754cffc6-99t9q"]
Sep 16 01:09:58.793319 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.793295    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"]
Sep 16 01:09:58.804177 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:09:58.804157    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-96c840/embedding-w0-96c840-kserve-7f57fc5cc6-wnkj2"]
Sep 16 01:10:00.583557 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:00.583484    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"]
Sep 16 01:10:00.583983 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:00.583731    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy" containerID="cri-o://d7e425093b7f57ac7d0a26b509cbcb500e08e6fbf91f5baefce54fab02711e02" gracePeriod=30
Sep 16 01:10:00.772674 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:00.772641    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4976c5c6-a606-43ec-b334-edd7b718a053" path="/var/lib/kubelet/pods/4976c5c6-a606-43ec-b334-edd7b718a053/volumes"
Sep 16 01:10:00.773053 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:00.773040    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" path="/var/lib/kubelet/pods/eee95b02-5eb5-4bd0-bdc2-6c7610dfb375/volumes"
Sep 16 01:10:05.803582 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.803552    2573 generic.go:356] "Generic (PLEG): container finished" podID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerID="d7e425093b7f57ac7d0a26b509cbcb500e08e6fbf91f5baefce54fab02711e02" exitCode=0
Sep 16 01:10:05.803960 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.803594    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" event={"ID":"06460b71-72f9-4fd3-93f8-d61f6ecd5be0","Type":"ContainerDied","Data":"d7e425093b7f57ac7d0a26b509cbcb500e08e6fbf91f5baefce54fab02711e02"}
Sep 16 01:10:05.803960 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.803649    2573 scope.go:117] "RemoveContainer" containerID="26554cfce7e1999be01a3336217853dd9578d2c4b2bd178618ad14513a11075d"
Sep 16 01:10:05.848306 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.848232    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:10:05.984695 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.984663    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istiod-ca-cert\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.984867 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.984725    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-envoy\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.984919 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.984892    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-data\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.984959 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.984941    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-podinfo\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.984991 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.984970    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-token\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.985039 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985021    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:10:05.985090 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985024    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-certs\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.985133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985084    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-socket\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.985305 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985180    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-data" (OuterVolumeSpecName: "istio-data") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.985305 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985234    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.985430 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985319    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ppwv\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-kube-api-access-8ppwv\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.985430 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985336    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.985430 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985396    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-credential-socket\") pod \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\" (UID: \"06460b71-72f9-4fd3-93f8-d61f6ecd5be0\") "
Sep 16 01:10:05.985957 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.985865    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.986162 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.986026    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.986162 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.986047    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.986162 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.986063    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.986162 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.986077    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.986162 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.986089    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:05.987313 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.987285    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-token" (OuterVolumeSpecName: "istio-token") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:10:05.987313 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.987302    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:10:05.987313 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.987316    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:10:05.987559 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:05.987544    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-kube-api-access-8ppwv" (OuterVolumeSpecName: "kube-api-access-8ppwv") pod "06460b71-72f9-4fd3-93f8-d61f6ecd5be0" (UID: "06460b71-72f9-4fd3-93f8-d61f6ecd5be0"). InnerVolumeSpecName "kube-api-access-8ppwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:10:06.087154 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.087120    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ppwv\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-kube-api-access-8ppwv\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:06.087154 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.087150    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:06.087154 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.087159    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:06.087381 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.087167    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06460b71-72f9-4fd3-93f8-d61f6ecd5be0-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:10:06.810734 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.810707    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"
Sep 16 01:10:06.811107 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.810753    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv" event={"ID":"06460b71-72f9-4fd3-93f8-d61f6ecd5be0","Type":"ContainerDied","Data":"3de7da098a4f28d739bccdf8011a31f0e1e93f034effa03932c45372cfaa1107"}
Sep 16 01:10:06.811107 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.810823    2573 scope.go:117] "RemoveContainer" containerID="d7e425093b7f57ac7d0a26b509cbcb500e08e6fbf91f5baefce54fab02711e02"
Sep 16 01:10:06.828682 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.828657    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"]
Sep 16 01:10:06.835074 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:06.835051    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-96c840-openshift-default-6c66cf578d-f8sbv"]
Sep 16 01:10:08.771249 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:08.771222    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" path="/var/lib/kubelet/pods/06460b71-72f9-4fd3-93f8-d61f6ecd5be0/volumes"
Sep 16 01:10:13.513343 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.513305    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"]
Sep 16 01:10:13.514022 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514002    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" containerName="main"
Sep 16 01:10:13.514104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514024    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" containerName="main"
Sep 16 01:10:13.514104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514043    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e5d02e-293f-49a4-8b63-038e112003f2" containerName="payload-processing"
Sep 16 01:10:13.514104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514052    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e5d02e-293f-49a4-8b63-038e112003f2" containerName="payload-processing"
Sep 16 01:10:13.514104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514069    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902400d2-31de-421c-8d5a-04b9c284c078" containerName="revoke-keys"
Sep 16 01:10:13.514104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514077    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="902400d2-31de-421c-8d5a-04b9c284c078" containerName="revoke-keys"
Sep 16 01:10:13.514104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514087    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4976c5c6-a606-43ec-b334-edd7b718a053" containerName="main"
Sep 16 01:10:13.514104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514095    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4976c5c6-a606-43ec-b334-edd7b718a053" containerName="main"
Sep 16 01:10:13.514104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514106    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy"
Sep 16 01:10:13.514378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514115    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy"
Sep 16 01:10:13.514378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514128    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy"
Sep 16 01:10:13.514378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514136    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy"
Sep 16 01:10:13.514378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514238    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy"
Sep 16 01:10:13.514378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514251    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e5d02e-293f-49a4-8b63-038e112003f2" containerName="payload-processing"
Sep 16 01:10:13.514378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514265    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="902400d2-31de-421c-8d5a-04b9c284c078" containerName="revoke-keys"
Sep 16 01:10:13.514378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514272    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee95b02-5eb5-4bd0-bdc2-6c7610dfb375" containerName="main"
Sep 16 01:10:13.514378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.514282    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4976c5c6-a606-43ec-b334-edd7b718a053" containerName="main"
Sep 16 01:10:13.519349 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.519328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.521705 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.521683    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f39e09-openshift-default-dockercfg-gvks4\""
Sep 16 01:10:13.527073 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.527048    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"]
Sep 16 01:10:13.665769 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.665728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-certs\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.665954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.665785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-podinfo\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.665954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.665837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-credential-socket\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.665954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.665866    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-data\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.665954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.665907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-token\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.665954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.665933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-envoy\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.665954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.665951    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cba585-f20d-48aa-8be8-ee6e78974664-istiod-ca-cert\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.666201 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.665972    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-socket\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.666201 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.666018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgp2v\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-kube-api-access-jgp2v\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.767466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-podinfo\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.767466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767434    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-credential-socket\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.767727 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767472    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-data\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.767727 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-token\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.767727 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767581    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-envoy\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.767727 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cba585-f20d-48aa-8be8-ee6e78974664-istiod-ca-cert\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.767727 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767638    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-socket\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.767727 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgp2v\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-kube-api-access-jgp2v\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.768053 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767862    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-credential-socket\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.768053 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767883    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-data\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.768053 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.767910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-certs\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.768222 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.768114    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-socket\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.768222 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.768210    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-certs\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.768383 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.768363    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cba585-f20d-48aa-8be8-ee6e78974664-istiod-ca-cert\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.770006 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.769985    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-envoy\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.770246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.770226    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-podinfo\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.775247 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.775221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-token\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.775470 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.775453    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgp2v\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-kube-api-access-jgp2v\") pod \"e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") " pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:13.832749 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:13.832719    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:14.116115 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:14.116063    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"]
Sep 16 01:10:14.118592 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:10:14.118562    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cba585_f20d_48aa_8be8_ee6e78974664.slice/crio-d5d089236d366ef98fc890f47d18e548de77e30e10fedc3506a1bf0afeac47ef WatchSource:0}: Error finding container d5d089236d366ef98fc890f47d18e548de77e30e10fedc3506a1bf0afeac47ef: Status 404 returned error can't find the container with id d5d089236d366ef98fc890f47d18e548de77e30e10fedc3506a1bf0afeac47ef
Sep 16 01:10:14.120935 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:14.120889    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:10:14.121015 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:14.120994    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:10:14.121056 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:14.121044    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:10:14.864079 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:14.864045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" event={"ID":"c9cba585-f20d-48aa-8be8-ee6e78974664","Type":"ContainerStarted","Data":"b8c476e4ad31ac8cf0a20617102328a29ab14dfde1d9445611fa60c5c944da2b"}
Sep 16 01:10:14.864462 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:14.864108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" event={"ID":"c9cba585-f20d-48aa-8be8-ee6e78974664","Type":"ContainerStarted","Data":"d5d089236d366ef98fc890f47d18e548de77e30e10fedc3506a1bf0afeac47ef"}
Sep 16 01:10:14.883469 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:14.883393    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podStartSLOduration=1.883372483 podStartE2EDuration="1.883372483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:10:14.882789582 +0000 UTC m=+1600.585611777" watchObservedRunningTime="2026-09-16 01:10:14.883372483 +0000 UTC m=+1600.586194679"
Sep 16 01:10:15.833779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:15.833747    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:15.840030 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:15.840007    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:15.870152 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:15.870124    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:15.870918 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:15.870902    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:24.767940 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:24.767912    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:10:31.911781 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:31.911732    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 01:10:31.912210 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:31.911812    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 01:10:34.994066 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:34.994038    2573 generic.go:356] "Generic (PLEG): container finished" podID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerID="b8c476e4ad31ac8cf0a20617102328a29ab14dfde1d9445611fa60c5c944da2b" exitCode=0
Sep 16 01:10:34.994395 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:34.994095    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" event={"ID":"c9cba585-f20d-48aa-8be8-ee6e78974664","Type":"ContainerDied","Data":"b8c476e4ad31ac8cf0a20617102328a29ab14dfde1d9445611fa60c5c944da2b"}
Sep 16 01:10:34.994499 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:34.994484    2573 scope.go:117] "RemoveContainer" containerID="b8c476e4ad31ac8cf0a20617102328a29ab14dfde1d9445611fa60c5c944da2b"
Sep 16 01:10:34.995498 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:34.995471    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:10:34.995587 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:34.995573    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:10:34.995629 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:34.995612    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:10:36.001965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:36.001931    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" event={"ID":"c9cba585-f20d-48aa-8be8-ee6e78974664","Type":"ContainerStarted","Data":"e175a257c52da91b4b7a618362b809ea7adcdfb8273db68ca6034c12d769fd89"}
Sep 16 01:10:36.833901 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:36.833865    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:36.834187 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:36.834158    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:10:36.834291 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:36.834249    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 01:10:37.833617 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:37.833579    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:10:37.834196 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:37.834163    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 01:10:38.833843 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:38.833808    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:10:38.834459 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:38.834428    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 01:10:39.833317 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:39.833271    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:10:39.833536 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:39.833346    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 01:10:40.834104 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:40.834070    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 01:10:40.834709 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:40.834677    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 01:10:41.837873 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:41.837844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:41.838265 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:41.838229    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:41.838942 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:41.838924    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:10:47.685198 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.685164    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"]
Sep 16 01:10:47.685708 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.685694    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="06460b71-72f9-4fd3-93f8-d61f6ecd5be0" containerName="istio-proxy"
Sep 16 01:10:47.687870 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.687853    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:47.690075 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.690054    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-45446b-dockercfg-xqq6l\""
Sep 16 01:10:47.690223 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.690200    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-45446b\""
Sep 16 01:10:47.698210 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.698191    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"]
Sep 16 01:10:47.815746 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.815716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r95rb\" (UniqueName: \"kubernetes.io/projected/39c8d413-1fa0-4364-8b62-751d4603d1a6-kube-api-access-r95rb\") pod \"payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s\" (UID: \"39c8d413-1fa0-4364-8b62-751d4603d1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:47.815916 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.815766    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c8d413-1fa0-4364-8b62-751d4603d1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s\" (UID: \"39c8d413-1fa0-4364-8b62-751d4603d1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:47.917125 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.917091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r95rb\" (UniqueName: \"kubernetes.io/projected/39c8d413-1fa0-4364-8b62-751d4603d1a6-kube-api-access-r95rb\") pod \"payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s\" (UID: \"39c8d413-1fa0-4364-8b62-751d4603d1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:47.917262 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.917136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c8d413-1fa0-4364-8b62-751d4603d1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s\" (UID: \"39c8d413-1fa0-4364-8b62-751d4603d1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:47.917894 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.917874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c8d413-1fa0-4364-8b62-751d4603d1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s\" (UID: \"39c8d413-1fa0-4364-8b62-751d4603d1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:47.924639 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.924616    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r95rb\" (UniqueName: \"kubernetes.io/projected/39c8d413-1fa0-4364-8b62-751d4603d1a6-kube-api-access-r95rb\") pod \"payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s\" (UID: \"39c8d413-1fa0-4364-8b62-751d4603d1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:47.999486 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:47.999465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:48.273685 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:48.273655    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"]
Sep 16 01:10:48.275396 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:10:48.275368    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c8d413_1fa0_4364_8b62_751d4603d1a6.slice/crio-56ab02f37a67f965d6ff3c3af06eb29d80efb7f9c68befd98cfb88134dcc9147 WatchSource:0}: Error finding container 56ab02f37a67f965d6ff3c3af06eb29d80efb7f9c68befd98cfb88134dcc9147: Status 404 returned error can't find the container with id 56ab02f37a67f965d6ff3c3af06eb29d80efb7f9c68befd98cfb88134dcc9147
Sep 16 01:10:49.086332 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:49.086297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s" event={"ID":"39c8d413-1fa0-4364-8b62-751d4603d1a6","Type":"ContainerStarted","Data":"a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c"}
Sep 16 01:10:49.086718 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:49.086359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s" event={"ID":"39c8d413-1fa0-4364-8b62-751d4603d1a6","Type":"ContainerStarted","Data":"56ab02f37a67f965d6ff3c3af06eb29d80efb7f9c68befd98cfb88134dcc9147"}
Sep 16 01:10:49.086718 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:49.086389    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:10:49.102635 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:49.102188    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s" podStartSLOduration=2.102169296 podStartE2EDuration="2.102169296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:10:49.102135392 +0000 UTC m=+1634.804957588" watchObservedRunningTime="2026-09-16 01:10:49.102169296 +0000 UTC m=+1634.804991492"
Sep 16 01:10:59.765496 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:10:59.765465    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:11:00.094623 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:00.094549    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:11:19.883233 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:19.883193    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"]
Sep 16 01:11:19.886107 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:19.886091    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:19.888263 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:19.888244    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e7d6db19-openshift-default-dockercfg-2cnmz\""
Sep 16 01:11:19.897691 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:19.897667    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"]
Sep 16 01:11:20.027552 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-credential-socket\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.027752 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-podinfo\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.027752 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-certs\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.027752 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-socket\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.027752 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istiod-ca-cert\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.027752 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-envoy\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.027966 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8m8\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-kube-api-access-5d8m8\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.027966 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-data\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.027966 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.027893    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-token\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.128744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.128713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-podinfo\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.128744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.128748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-certs\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.128994 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.128801    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-socket\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.128994 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.128839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istiod-ca-cert\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.128994 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.128874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-envoy\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.128994 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.128968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8m8\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-kube-api-access-5d8m8\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.129215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.128999    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-data\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.129215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.129026    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-token\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.129215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.129093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-credential-socket\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.129392 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.129294    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-socket\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.129453 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.129425    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-credential-socket\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.129509 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.129486    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-data\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.129700 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.129678    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istiod-ca-cert\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.129805 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.129781    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-certs\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.131865 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.131843    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-envoy\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.131965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.131873    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-podinfo\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.136367 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.136313    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-token\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.137409 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.137388    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8m8\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-kube-api-access-5d8m8\") pod \"e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") " pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.198543 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.198500    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:20.476285 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.476261    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"]
Sep 16 01:11:20.478185 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:11:20.478159    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ea7b01_a491_46d7_bf42_35eb2bd41463.slice/crio-8e4429aedaf4bde8ad2c3d17f1d24f3fcf2aa728c5d439d2b3b3a4992369a397 WatchSource:0}: Error finding container 8e4429aedaf4bde8ad2c3d17f1d24f3fcf2aa728c5d439d2b3b3a4992369a397: Status 404 returned error can't find the container with id 8e4429aedaf4bde8ad2c3d17f1d24f3fcf2aa728c5d439d2b3b3a4992369a397
Sep 16 01:11:20.480508 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.480475    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:11:20.480644 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.480556    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:11:20.480644 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:20.480588    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:11:21.278126 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:21.278086    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" event={"ID":"a2ea7b01-a491-46d7-bf42-35eb2bd41463","Type":"ContainerStarted","Data":"3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229"}
Sep 16 01:11:21.278700 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:21.278145    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" event={"ID":"a2ea7b01-a491-46d7-bf42-35eb2bd41463","Type":"ContainerStarted","Data":"8e4429aedaf4bde8ad2c3d17f1d24f3fcf2aa728c5d439d2b3b3a4992369a397"}
Sep 16 01:11:21.298777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:21.298729    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" podStartSLOduration=2.298713536 podStartE2EDuration="2.298713536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:21.296504715 +0000 UTC m=+1666.999326911" watchObservedRunningTime="2026-09-16 01:11:21.298713536 +0000 UTC m=+1667.001535731"
Sep 16 01:11:22.199119 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:22.199086    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:22.204584 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:22.204561    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:22.284511 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:22.284481    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:22.285556 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:22.285512    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:24.801735 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:24.801707    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"]
Sep 16 01:11:26.509241 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.509209    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"]
Sep 16 01:11:26.511786 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.511770    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:26.513903 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.513880    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e7d6db19\""
Sep 16 01:11:26.521486 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.521459    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"]
Sep 16 01:11:26.597102 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.597075    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw\" (UID: \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:26.597239 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.597119    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2jk\" (UniqueName: \"kubernetes.io/projected/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-kube-api-access-rl2jk\") pod \"payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw\" (UID: \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:26.698087 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.698050    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw\" (UID: \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:26.698294 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.698119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2jk\" (UniqueName: \"kubernetes.io/projected/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-kube-api-access-rl2jk\") pod \"payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw\" (UID: \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:26.698853 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.698828    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw\" (UID: \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:26.705904 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.705882    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2jk\" (UniqueName: \"kubernetes.io/projected/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-kube-api-access-rl2jk\") pod \"payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw\" (UID: \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:26.823313 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:26.823246    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:27.101641 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:27.101611    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"]
Sep 16 01:11:27.104358 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:11:27.104326    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bb2c59_f4ff_4cce_b8f2_5080e8112fb5.slice/crio-4883fca0be6e88b8e2c814b8d848563e14ce91cb017eb520bf854923fbbf9fa1 WatchSource:0}: Error finding container 4883fca0be6e88b8e2c814b8d848563e14ce91cb017eb520bf854923fbbf9fa1: Status 404 returned error can't find the container with id 4883fca0be6e88b8e2c814b8d848563e14ce91cb017eb520bf854923fbbf9fa1
Sep 16 01:11:27.318040 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:27.318009    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw" event={"ID":"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5","Type":"ContainerStarted","Data":"9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb"}
Sep 16 01:11:27.318208 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:27.318063    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw" event={"ID":"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5","Type":"ContainerStarted","Data":"4883fca0be6e88b8e2c814b8d848563e14ce91cb017eb520bf854923fbbf9fa1"}
Sep 16 01:11:27.318208 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:27.318135    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:27.333878 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:27.333824    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw" podStartSLOduration=1.333805785 podStartE2EDuration="1.333805785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:27.332695576 +0000 UTC m=+1673.035517770" watchObservedRunningTime="2026-09-16 01:11:27.333805785 +0000 UTC m=+1673.036627981"
Sep 16 01:11:34.948143 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:34.948109    2573 scope.go:117] "RemoveContainer" containerID="67c8ae656d6b59dd9c1dedbf6248de23d54482e2c9fdb5b4e80a544bc86da665"
Sep 16 01:11:34.957714 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:34.957689    2573 scope.go:117] "RemoveContainer" containerID="d77dd8acc75e3803a7e7b9ba79c7c290ac21d57350bd2486688054a455a0bf4f"
Sep 16 01:11:36.765738 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:36.765704    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:11:38.325672 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:38.325642    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:11:39.822508 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:39.822462    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk" podUID="4ceb9338-dcf1-4c85-885d-16d3f2c546cb" containerName="main" containerID="cri-o://f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535" gracePeriod=45
Sep 16 01:11:40.123207 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.123187    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:11:40.266266 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266231    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdrkq\" (UniqueName: \"kubernetes.io/projected/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-kube-api-access-hdrkq\") pod \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") "
Sep 16 01:11:40.266413 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266298    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-home\") pod \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") "
Sep 16 01:11:40.266469 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266407    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tmp-dir\") pod \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") "
Sep 16 01:11:40.266512 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266496    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-dshm\") pod \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") "
Sep 16 01:11:40.266512 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266502    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-home" (OuterVolumeSpecName: "home") pod "4ceb9338-dcf1-4c85-885d-16d3f2c546cb" (UID: "4ceb9338-dcf1-4c85-885d-16d3f2c546cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:11:40.266626 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266559    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-model-cache\") pod \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") "
Sep 16 01:11:40.266626 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266594    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tls-certs\") pod \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\" (UID: \"4ceb9338-dcf1-4c85-885d-16d3f2c546cb\") "
Sep 16 01:11:40.266728 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266649    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ceb9338-dcf1-4c85-885d-16d3f2c546cb" (UID: "4ceb9338-dcf1-4c85-885d-16d3f2c546cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:11:40.266846 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266810    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ceb9338-dcf1-4c85-885d-16d3f2c546cb" (UID: "4ceb9338-dcf1-4c85-885d-16d3f2c546cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:11:40.266978 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266889    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:11:40.267093 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.266991    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:11:40.268779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.268747    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ceb9338-dcf1-4c85-885d-16d3f2c546cb" (UID: "4ceb9338-dcf1-4c85-885d-16d3f2c546cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:11:40.268779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.268753    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-kube-api-access-hdrkq" (OuterVolumeSpecName: "kube-api-access-hdrkq") pod "4ceb9338-dcf1-4c85-885d-16d3f2c546cb" (UID: "4ceb9338-dcf1-4c85-885d-16d3f2c546cb"). InnerVolumeSpecName "kube-api-access-hdrkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:11:40.268779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.268747    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-dshm" (OuterVolumeSpecName: "dshm") pod "4ceb9338-dcf1-4c85-885d-16d3f2c546cb" (UID: "4ceb9338-dcf1-4c85-885d-16d3f2c546cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:11:40.368163 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.368080    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:11:40.368163 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.368108    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:11:40.368163 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.368118    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:11:40.368163 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.368128    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdrkq\" (UniqueName: \"kubernetes.io/projected/4ceb9338-dcf1-4c85-885d-16d3f2c546cb-kube-api-access-hdrkq\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:11:40.398264 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.398232    2573 generic.go:356] "Generic (PLEG): container finished" podID="4ceb9338-dcf1-4c85-885d-16d3f2c546cb" containerID="f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535" exitCode=0
Sep 16 01:11:40.398411 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.398286    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"
Sep 16 01:11:40.398411 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.398297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk" event={"ID":"4ceb9338-dcf1-4c85-885d-16d3f2c546cb","Type":"ContainerDied","Data":"f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535"}
Sep 16 01:11:40.398411 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.398330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk" event={"ID":"4ceb9338-dcf1-4c85-885d-16d3f2c546cb","Type":"ContainerDied","Data":"87008a3123f36f61839af70dcaa2969d547f1f26011b67f66334ecfc38a8430f"}
Sep 16 01:11:40.398411 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.398347    2573 scope.go:117] "RemoveContainer" containerID="f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535"
Sep 16 01:11:40.408210 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.408194    2573 scope.go:117] "RemoveContainer" containerID="f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535"
Sep 16 01:11:40.408510 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:11:40.408486    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535\": container with ID starting with f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535 not found: ID does not exist" containerID="f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535"
Sep 16 01:11:40.408604 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.408542    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535"} err="failed to get container status \"f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535\": rpc error: code = NotFound desc = could not find container \"f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535\": container with ID starting with f716bd93ae649b2094d188d0fac6a4d085df5f9c144060bb270487bd02484535 not found: ID does not exist"
Sep 16 01:11:40.418876 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.418855    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"]
Sep 16 01:11:40.424178 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.424154    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/auth-test-model-0ee1bf-kserve-5ddd548f47-8rddk"]
Sep 16 01:11:40.773161 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:40.773123    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ceb9338-dcf1-4c85-885d-16d3f2c546cb" path="/var/lib/kubelet/pods/4ceb9338-dcf1-4c85-885d-16d3f2c546cb/volumes"
Sep 16 01:11:44.424390 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:44.424304    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerID="3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229" exitCode=0
Sep 16 01:11:44.424390 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:44.424378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" event={"ID":"a2ea7b01-a491-46d7-bf42-35eb2bd41463","Type":"ContainerDied","Data":"3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229"}
Sep 16 01:11:44.424821 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:44.424812    2573 scope.go:117] "RemoveContainer" containerID="3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229"
Sep 16 01:11:44.425978 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:44.425954    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:11:44.426041 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:44.426024    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:11:44.426082 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:44.426056    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:11:45.434570 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:45.434515    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" event={"ID":"a2ea7b01-a491-46d7-bf42-35eb2bd41463","Type":"ContainerStarted","Data":"1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a"}
Sep 16 01:11:46.198793 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:46.198760    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:46.198965 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:46.198902    2573 patch_prober.go:28] interesting pod/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 16 01:11:46.199014 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:46.198967    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 01:11:47.199163 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:47.199127    2573 patch_prober.go:28] interesting pod/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 16 01:11:47.199575 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:47.199189    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 01:11:48.199624 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:48.199581    2573 patch_prober.go:28] interesting pod/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 16 01:11:48.200020 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:48.199649    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 01:11:49.198886 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:49.198853    2573 patch_prober.go:28] interesting pod/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 16 01:11:49.199062 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:49.198916    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 01:11:50.199139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:50.199101    2573 patch_prober.go:28] interesting pod/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 16 01:11:50.199625 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:50.199173    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 01:11:51.203633 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:51.203600    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:51.204024 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:51.203900    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:51.204697 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:51.204681    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:11:53.878876 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.878837    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"]
Sep 16 01:11:53.879438 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.879420    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ceb9338-dcf1-4c85-885d-16d3f2c546cb" containerName="main"
Sep 16 01:11:53.879568 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.879446    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ceb9338-dcf1-4c85-885d-16d3f2c546cb" containerName="main"
Sep 16 01:11:53.879633 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.879593    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ceb9338-dcf1-4c85-885d-16d3f2c546cb" containerName="main"
Sep 16 01:11:53.884159 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.884102    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:53.887040 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.887018    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e7d6db19\"/\"default-dockercfg-z555x\""
Sep 16 01:11:53.887220 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.887031    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e7d6db19\"/\"e2e-lifecycle-model-e7d6db19-kserve-self-signed-certs\""
Sep 16 01:11:53.887308 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.887079    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e7d6db19\"/\"openshift-service-ca.crt\""
Sep 16 01:11:53.887308 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.887101    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e7d6db19\"/\"kube-root-ca.crt\""
Sep 16 01:11:53.890701 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.890676    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"]
Sep 16 01:11:53.922166 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.922142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-tmp-dir\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:53.922282 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.922176    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47606ab-0865-48fe-a2f8-101237cdee84-tls-certs\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:53.922282 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.922196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-model-cache\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:53.922282 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.922213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdq4\" (UniqueName: \"kubernetes.io/projected/f47606ab-0865-48fe-a2f8-101237cdee84-kube-api-access-qmdq4\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:53.922409 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.922281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-home\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:53.922409 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:53.922391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-dshm\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023340 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-tmp-dir\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023350    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47606ab-0865-48fe-a2f8-101237cdee84-tls-certs\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-model-cache\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdq4\" (UniqueName: \"kubernetes.io/projected/f47606ab-0865-48fe-a2f8-101237cdee84-kube-api-access-qmdq4\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-home\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-dshm\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023824 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023797    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-tmp-dir\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023885 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-model-cache\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.023951 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.023914    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-home\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.026022 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.025997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-dshm\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.026271 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.026253    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47606ab-0865-48fe-a2f8-101237cdee84-tls-certs\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.030874 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.030854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdq4\" (UniqueName: \"kubernetes.io/projected/f47606ab-0865-48fe-a2f8-101237cdee84-kube-api-access-qmdq4\") pod \"e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") " pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.197065 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.197040    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:11:54.473649 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.473622    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"]
Sep 16 01:11:54.474998 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:11:54.474970    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf47606ab_0865_48fe_a2f8_101237cdee84.slice/crio-cf54dacd863d6e915b1ef9ce726e00283ed7e346c86328b35849748713997fb5 WatchSource:0}: Error finding container cf54dacd863d6e915b1ef9ce726e00283ed7e346c86328b35849748713997fb5: Status 404 returned error can't find the container with id cf54dacd863d6e915b1ef9ce726e00283ed7e346c86328b35849748713997fb5
Sep 16 01:11:54.498550 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:54.498502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx" event={"ID":"f47606ab-0865-48fe-a2f8-101237cdee84","Type":"ContainerStarted","Data":"cf54dacd863d6e915b1ef9ce726e00283ed7e346c86328b35849748713997fb5"}
Sep 16 01:11:55.506033 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:55.505990    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx" event={"ID":"f47606ab-0865-48fe-a2f8-101237cdee84","Type":"ContainerStarted","Data":"d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b"}
Sep 16 01:11:55.521893 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:11:55.521844    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx" podStartSLOduration=2.521831079 podStartE2EDuration="2.521831079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:11:55.520471165 +0000 UTC m=+1701.223293361" watchObservedRunningTime="2026-09-16 01:11:55.521831079 +0000 UTC m=+1701.224653274"
Sep 16 01:12:00.765389 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:00.765356    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:12:04.197992 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:04.197959    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:12:04.198385 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:04.198218    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:12:04.210874 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:04.210844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:12:04.571278 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:04.571200    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:12:07.185302 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.185270    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"]
Sep 16 01:12:07.189055 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.189038    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.191393 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.191370    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"kube-root-ca.crt\""
Sep 16 01:12:07.192074 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.192055    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"default-dockercfg-rnjz5\""
Sep 16 01:12:07.192199 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.192090    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"openshift-service-ca.crt\""
Sep 16 01:12:07.192199 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.192056    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-0ee1bf\"/\"sub-test-model-0ee1bf-kserve-self-signed-certs\""
Sep 16 01:12:07.197724 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.197704    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"]
Sep 16 01:12:07.251089 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.251058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-home\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.251242 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.251126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-dshm\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.251242 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.251173    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-model-cache\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.251365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.251262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrxk\" (UniqueName: \"kubernetes.io/projected/eb70c855-2622-4b0e-8229-58bc9085d0ed-kube-api-access-wvrxk\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.251365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.251328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-tmp-dir\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.251468 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.251373    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb70c855-2622-4b0e-8229-58bc9085d0ed-tls-certs\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.352607 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.352572    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-dshm\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.352607 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.352612    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-model-cache\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.352852 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.352647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrxk\" (UniqueName: \"kubernetes.io/projected/eb70c855-2622-4b0e-8229-58bc9085d0ed-kube-api-access-wvrxk\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.352852 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.352699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-tmp-dir\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.352852 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.352735    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb70c855-2622-4b0e-8229-58bc9085d0ed-tls-certs\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.352852 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.352810    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-home\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.353117 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.353092    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-model-cache\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.353180 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.353111    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-tmp-dir\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.353180 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.353141    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-home\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.355006 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.354979    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-dshm\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.355321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.355304    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb70c855-2622-4b0e-8229-58bc9085d0ed-tls-certs\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.359618 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.359599    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrxk\" (UniqueName: \"kubernetes.io/projected/eb70c855-2622-4b0e-8229-58bc9085d0ed-kube-api-access-wvrxk\") pod \"sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") " pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.501809 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.501774    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:07.778359 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:07.778321    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"]
Sep 16 01:12:07.779720 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:12:07.779696    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb70c855_2622_4b0e_8229_58bc9085d0ed.slice/crio-958757da59692440bd4b9348759823775b0871dda1af54821531a0b0d83d5b8d WatchSource:0}: Error finding container 958757da59692440bd4b9348759823775b0871dda1af54821531a0b0d83d5b8d: Status 404 returned error can't find the container with id 958757da59692440bd4b9348759823775b0871dda1af54821531a0b0d83d5b8d
Sep 16 01:12:08.585084 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:08.585039    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8" event={"ID":"eb70c855-2622-4b0e-8229-58bc9085d0ed","Type":"ContainerStarted","Data":"b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3"}
Sep 16 01:12:08.585456 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:08.585095    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8" event={"ID":"eb70c855-2622-4b0e-8229-58bc9085d0ed","Type":"ContainerStarted","Data":"958757da59692440bd4b9348759823775b0871dda1af54821531a0b0d83d5b8d"}
Sep 16 01:12:08.601293 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:08.601232    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8" podStartSLOduration=1.601213815 podStartE2EDuration="1.601213815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:12:08.600986101 +0000 UTC m=+1714.303808317" watchObservedRunningTime="2026-09-16 01:12:08.601213815 +0000 UTC m=+1714.304036013"
Sep 16 01:12:17.502204 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:17.502164    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:17.502788 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:17.502256    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:17.514949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:17.514923    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:17.650224 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:17.650192    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:12:34.984943 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:34.984904    2573 scope.go:117] "RemoveContainer" containerID="2ca4c3c9e6db43e328673693b8fc718506710c8821e9f3eab46dca0347c67f87"
Sep 16 01:12:34.994336 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:34.994313    2573 scope.go:117] "RemoveContainer" containerID="289eeeae77b824680e220b737d7a518e490a429766829e4032c297fb253b176d"
Sep 16 01:12:38.438939 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.438908    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"]
Sep 16 01:12:38.442282 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.442264    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"
Sep 16 01:12:38.444274 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.444252    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:12:38.444385 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.444321    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:12:38.448623 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.448603    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"]
Sep 16 01:12:38.569260 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.569217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d961a-1482-48fc-bd97-c39e95b5d040-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e7d6db19-z766c\" (UID: \"c66d961a-1482-48fc-bd97-c39e95b5d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"
Sep 16 01:12:38.670547 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.670485    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d961a-1482-48fc-bd97-c39e95b5d040-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e7d6db19-z766c\" (UID: \"c66d961a-1482-48fc-bd97-c39e95b5d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"
Sep 16 01:12:38.671170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.671150    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d961a-1482-48fc-bd97-c39e95b5d040-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e7d6db19-z766c\" (UID: \"c66d961a-1482-48fc-bd97-c39e95b5d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"
Sep 16 01:12:38.755662 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:38.755633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"
Sep 16 01:12:39.029703 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:39.029676    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"]
Sep 16 01:12:39.032309 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:12:39.032276    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66d961a_1482_48fc_bd97_c39e95b5d040.slice/crio-fec5520cfe9f8cb8e414ec1d0f0ac3672770afbe204e7c30eb9fd38b633f6b9a WatchSource:0}: Error finding container fec5520cfe9f8cb8e414ec1d0f0ac3672770afbe204e7c30eb9fd38b633f6b9a: Status 404 returned error can't find the container with id fec5520cfe9f8cb8e414ec1d0f0ac3672770afbe204e7c30eb9fd38b633f6b9a
Sep 16 01:12:39.767779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:39.767743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c" event={"ID":"c66d961a-1482-48fc-bd97-c39e95b5d040","Type":"ContainerStarted","Data":"f7b5386b0af581237b9253724ffe25eb921fde7616739c5befce92995db0d39a"}
Sep 16 01:12:39.768202 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:39.767794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c" event={"ID":"c66d961a-1482-48fc-bd97-c39e95b5d040","Type":"ContainerStarted","Data":"fec5520cfe9f8cb8e414ec1d0f0ac3672770afbe204e7c30eb9fd38b633f6b9a"}
Sep 16 01:12:39.787004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:39.786956    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c" podStartSLOduration=1.786939554 podStartE2EDuration="1.786939554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:12:39.784928257 +0000 UTC m=+1745.487750450" watchObservedRunningTime="2026-09-16 01:12:39.786939554 +0000 UTC m=+1745.489761749"
Sep 16 01:12:40.774714 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:40.774680    2573 generic.go:356] "Generic (PLEG): container finished" podID="c66d961a-1482-48fc-bd97-c39e95b5d040" containerID="f7b5386b0af581237b9253724ffe25eb921fde7616739c5befce92995db0d39a" exitCode=0
Sep 16 01:12:40.775089 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:40.774764    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c" event={"ID":"c66d961a-1482-48fc-bd97-c39e95b5d040","Type":"ContainerDied","Data":"f7b5386b0af581237b9253724ffe25eb921fde7616739c5befce92995db0d39a"}
Sep 16 01:12:41.928664 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:41.928642    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"
Sep 16 01:12:42.004490 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:42.004461    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d961a-1482-48fc-bd97-c39e95b5d040-maas-api-service-ca\") pod \"c66d961a-1482-48fc-bd97-c39e95b5d040\" (UID: \"c66d961a-1482-48fc-bd97-c39e95b5d040\") "
Sep 16 01:12:42.004869 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:42.004846    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66d961a-1482-48fc-bd97-c39e95b5d040-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c66d961a-1482-48fc-bd97-c39e95b5d040" (UID: "c66d961a-1482-48fc-bd97-c39e95b5d040"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:12:42.005008 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:42.004994    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d961a-1482-48fc-bd97-c39e95b5d040-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:12:42.788513 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:42.788487    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"
Sep 16 01:12:42.788513 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:42.788507    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c" event={"ID":"c66d961a-1482-48fc-bd97-c39e95b5d040","Type":"ContainerDied","Data":"fec5520cfe9f8cb8e414ec1d0f0ac3672770afbe204e7c30eb9fd38b633f6b9a"}
Sep 16 01:12:42.788708 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:42.788563    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fec5520cfe9f8cb8e414ec1d0f0ac3672770afbe204e7c30eb9fd38b633f6b9a"
Sep 16 01:12:54.078859 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.078822    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"]
Sep 16 01:12:54.079460 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.079052    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw" podUID="d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5" containerName="payload-pre-processing" containerID="cri-o://9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb" gracePeriod=30
Sep 16 01:12:54.261749 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.261718    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"]
Sep 16 01:12:54.265626 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.265598    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e7d6db19-z766c"]
Sep 16 01:12:54.340199 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.340128    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:12:54.439335 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.438504    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl2jk\" (UniqueName: \"kubernetes.io/projected/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-kube-api-access-rl2jk\") pod \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\" (UID: \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\") "
Sep 16 01:12:54.439335 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.438672    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-plugins-config-volume\") pod \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\" (UID: \"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5\") "
Sep 16 01:12:54.439825 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.439789    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5" (UID: "d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:12:54.442457 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.442416    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-kube-api-access-rl2jk" (OuterVolumeSpecName: "kube-api-access-rl2jk") pod "d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5" (UID: "d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5"). InnerVolumeSpecName "kube-api-access-rl2jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:12:54.540627 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.540594    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:12:54.540627 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.540621    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2jk\" (UniqueName: \"kubernetes.io/projected/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5-kube-api-access-rl2jk\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:12:54.776537 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.776485    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66d961a-1482-48fc-bd97-c39e95b5d040" path="/var/lib/kubelet/pods/c66d961a-1482-48fc-bd97-c39e95b5d040/volumes"
Sep 16 01:12:54.858586 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.858548    2573 generic.go:356] "Generic (PLEG): container finished" podID="d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5" containerID="9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb" exitCode=0
Sep 16 01:12:54.858586 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.858589    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"
Sep 16 01:12:54.858796 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.858626    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw" event={"ID":"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5","Type":"ContainerDied","Data":"9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb"}
Sep 16 01:12:54.858796 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.858675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw" event={"ID":"d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5","Type":"ContainerDied","Data":"4883fca0be6e88b8e2c814b8d848563e14ce91cb017eb520bf854923fbbf9fa1"}
Sep 16 01:12:54.858796 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.858697    2573 scope.go:117] "RemoveContainer" containerID="9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb"
Sep 16 01:12:54.868317 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.868301    2573 scope.go:117] "RemoveContainer" containerID="9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb"
Sep 16 01:12:54.868606 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:12:54.868587    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb\": container with ID starting with 9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb not found: ID does not exist" containerID="9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb"
Sep 16 01:12:54.868659 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.868617    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb"} err="failed to get container status \"9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb\": rpc error: code = NotFound desc = could not find container \"9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb\": container with ID starting with 9e8c5fab705742f75b1ca0e67ab4b2305dfdc1881ff9047de70adc61f05968bb not found: ID does not exist"
Sep 16 01:12:54.873530 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.873498    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"]
Sep 16 01:12:54.877507 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:54.877483    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7d6db19-77b958bf6b-cbwhw"]
Sep 16 01:12:56.770903 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:12:56.770865    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5" path="/var/lib/kubelet/pods/d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5/volumes"
Sep 16 01:13:02.569833 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:02.569798    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"]
Sep 16 01:13:06.765280 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:06.765242    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:13:10.697084 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:10.697046    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"]
Sep 16 01:13:17.589998 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.589941    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx" podUID="f47606ab-0865-48fe-a2f8-101237cdee84" containerName="main" containerID="cri-o://d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b" gracePeriod=45
Sep 16 01:13:17.853954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.853897    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:13:17.970076 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970045    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-model-cache\") pod \"f47606ab-0865-48fe-a2f8-101237cdee84\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") "
Sep 16 01:13:17.970233 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970095    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47606ab-0865-48fe-a2f8-101237cdee84-tls-certs\") pod \"f47606ab-0865-48fe-a2f8-101237cdee84\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") "
Sep 16 01:13:17.970233 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970128    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-tmp-dir\") pod \"f47606ab-0865-48fe-a2f8-101237cdee84\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") "
Sep 16 01:13:17.970233 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970195    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmdq4\" (UniqueName: \"kubernetes.io/projected/f47606ab-0865-48fe-a2f8-101237cdee84-kube-api-access-qmdq4\") pod \"f47606ab-0865-48fe-a2f8-101237cdee84\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") "
Sep 16 01:13:17.970361 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970243    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-dshm\") pod \"f47606ab-0865-48fe-a2f8-101237cdee84\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") "
Sep 16 01:13:17.970361 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970270    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-home\") pod \"f47606ab-0865-48fe-a2f8-101237cdee84\" (UID: \"f47606ab-0865-48fe-a2f8-101237cdee84\") "
Sep 16 01:13:17.970361 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970299    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-model-cache" (OuterVolumeSpecName: "model-cache") pod "f47606ab-0865-48fe-a2f8-101237cdee84" (UID: "f47606ab-0865-48fe-a2f8-101237cdee84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:17.970560 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970434    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f47606ab-0865-48fe-a2f8-101237cdee84" (UID: "f47606ab-0865-48fe-a2f8-101237cdee84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:17.970666 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970638    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-home" (OuterVolumeSpecName: "home") pod "f47606ab-0865-48fe-a2f8-101237cdee84" (UID: "f47606ab-0865-48fe-a2f8-101237cdee84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:17.970788 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970648    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:17.970788 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.970696    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:17.972429 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.972405    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-dshm" (OuterVolumeSpecName: "dshm") pod "f47606ab-0865-48fe-a2f8-101237cdee84" (UID: "f47606ab-0865-48fe-a2f8-101237cdee84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:17.972508 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.972487    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47606ab-0865-48fe-a2f8-101237cdee84-kube-api-access-qmdq4" (OuterVolumeSpecName: "kube-api-access-qmdq4") pod "f47606ab-0865-48fe-a2f8-101237cdee84" (UID: "f47606ab-0865-48fe-a2f8-101237cdee84"). InnerVolumeSpecName "kube-api-access-qmdq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:13:17.972579 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.972510    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47606ab-0865-48fe-a2f8-101237cdee84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f47606ab-0865-48fe-a2f8-101237cdee84" (UID: "f47606ab-0865-48fe-a2f8-101237cdee84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:13:17.996389 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.996363    2573 generic.go:356] "Generic (PLEG): container finished" podID="f47606ab-0865-48fe-a2f8-101237cdee84" containerID="d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b" exitCode=0
Sep 16 01:13:17.996497 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.996402    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"
Sep 16 01:13:17.996497 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.996441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx" event={"ID":"f47606ab-0865-48fe-a2f8-101237cdee84","Type":"ContainerDied","Data":"d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b"}
Sep 16 01:13:17.996497 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.996483    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx" event={"ID":"f47606ab-0865-48fe-a2f8-101237cdee84","Type":"ContainerDied","Data":"cf54dacd863d6e915b1ef9ce726e00283ed7e346c86328b35849748713997fb5"}
Sep 16 01:13:17.996656 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:17.996505    2573 scope.go:117] "RemoveContainer" containerID="d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b"
Sep 16 01:13:18.006234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.006218    2573 scope.go:117] "RemoveContainer" containerID="d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b"
Sep 16 01:13:18.006605 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:13:18.006565    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b\": container with ID starting with d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b not found: ID does not exist" containerID="d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b"
Sep 16 01:13:18.006605 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.006597    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b"} err="failed to get container status \"d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b\": rpc error: code = NotFound desc = could not find container \"d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b\": container with ID starting with d0e5bba2f62c7e286f639b667acb2f78981d43de808f3db42da9d10860f0165b not found: ID does not exist"
Sep 16 01:13:18.016274 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.016249    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"]
Sep 16 01:13:18.019486 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.019462    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e7d6db19/e2e-lifecycle-model-e7d6db19-kserve-7d4958bcf5-9h6rx"]
Sep 16 01:13:18.072151 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.072126    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmdq4\" (UniqueName: \"kubernetes.io/projected/f47606ab-0865-48fe-a2f8-101237cdee84-kube-api-access-qmdq4\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:18.072151 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.072149    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:18.072296 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.072158    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47606ab-0865-48fe-a2f8-101237cdee84-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:18.072296 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.072167    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47606ab-0865-48fe-a2f8-101237cdee84-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:18.771931 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:18.771899    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47606ab-0865-48fe-a2f8-101237cdee84" path="/var/lib/kubelet/pods/f47606ab-0865-48fe-a2f8-101237cdee84/volumes"
Sep 16 01:13:25.716853 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:25.716797    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8" podUID="eb70c855-2622-4b0e-8229-58bc9085d0ed" containerName="main" containerID="cri-o://b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3" gracePeriod=45
Sep 16 01:13:25.979803 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:25.979738    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:13:26.046065 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.046034    2573 generic.go:356] "Generic (PLEG): container finished" podID="eb70c855-2622-4b0e-8229-58bc9085d0ed" containerID="b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3" exitCode=0
Sep 16 01:13:26.046226 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.046076    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"
Sep 16 01:13:26.046226 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.046125    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8" event={"ID":"eb70c855-2622-4b0e-8229-58bc9085d0ed","Type":"ContainerDied","Data":"b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3"}
Sep 16 01:13:26.046226 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.046165    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8" event={"ID":"eb70c855-2622-4b0e-8229-58bc9085d0ed","Type":"ContainerDied","Data":"958757da59692440bd4b9348759823775b0871dda1af54821531a0b0d83d5b8d"}
Sep 16 01:13:26.046226 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.046182    2573 scope.go:117] "RemoveContainer" containerID="b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3"
Sep 16 01:13:26.055376 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.055358    2573 scope.go:117] "RemoveContainer" containerID="b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3"
Sep 16 01:13:26.055674 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:13:26.055654    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3\": container with ID starting with b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3 not found: ID does not exist" containerID="b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3"
Sep 16 01:13:26.055757 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.055688    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3"} err="failed to get container status \"b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3\": rpc error: code = NotFound desc = could not find container \"b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3\": container with ID starting with b217503631f87381ba5a5eaad297043a4736b3b8270ab2a08f7ff6a0ed134ba3 not found: ID does not exist"
Sep 16 01:13:26.145466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.145436    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-dshm\") pod \"eb70c855-2622-4b0e-8229-58bc9085d0ed\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") "
Sep 16 01:13:26.145615 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.145502    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb70c855-2622-4b0e-8229-58bc9085d0ed-tls-certs\") pod \"eb70c855-2622-4b0e-8229-58bc9085d0ed\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") "
Sep 16 01:13:26.145615 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.145585    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-home\") pod \"eb70c855-2622-4b0e-8229-58bc9085d0ed\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") "
Sep 16 01:13:26.145744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.145663    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-model-cache\") pod \"eb70c855-2622-4b0e-8229-58bc9085d0ed\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") "
Sep 16 01:13:26.145744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.145691    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvrxk\" (UniqueName: \"kubernetes.io/projected/eb70c855-2622-4b0e-8229-58bc9085d0ed-kube-api-access-wvrxk\") pod \"eb70c855-2622-4b0e-8229-58bc9085d0ed\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") "
Sep 16 01:13:26.145744 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.145722    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-tmp-dir\") pod \"eb70c855-2622-4b0e-8229-58bc9085d0ed\" (UID: \"eb70c855-2622-4b0e-8229-58bc9085d0ed\") "
Sep 16 01:13:26.145902 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.145848    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-home" (OuterVolumeSpecName: "home") pod "eb70c855-2622-4b0e-8229-58bc9085d0ed" (UID: "eb70c855-2622-4b0e-8229-58bc9085d0ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:26.145976 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.145948    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb70c855-2622-4b0e-8229-58bc9085d0ed" (UID: "eb70c855-2622-4b0e-8229-58bc9085d0ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:26.146129 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.146038    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb70c855-2622-4b0e-8229-58bc9085d0ed" (UID: "eb70c855-2622-4b0e-8229-58bc9085d0ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:26.146335 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.146307    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:26.146388 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.146333    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:26.147670 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.147650    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-dshm" (OuterVolumeSpecName: "dshm") pod "eb70c855-2622-4b0e-8229-58bc9085d0ed" (UID: "eb70c855-2622-4b0e-8229-58bc9085d0ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:26.147765 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.147728    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb70c855-2622-4b0e-8229-58bc9085d0ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb70c855-2622-4b0e-8229-58bc9085d0ed" (UID: "eb70c855-2622-4b0e-8229-58bc9085d0ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:13:26.147826 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.147754    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb70c855-2622-4b0e-8229-58bc9085d0ed-kube-api-access-wvrxk" (OuterVolumeSpecName: "kube-api-access-wvrxk") pod "eb70c855-2622-4b0e-8229-58bc9085d0ed" (UID: "eb70c855-2622-4b0e-8229-58bc9085d0ed"). InnerVolumeSpecName "kube-api-access-wvrxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:13:26.247141 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.247083    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:26.247141 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.247105    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb70c855-2622-4b0e-8229-58bc9085d0ed-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:26.247141 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.247115    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvrxk\" (UniqueName: \"kubernetes.io/projected/eb70c855-2622-4b0e-8229-58bc9085d0ed-kube-api-access-wvrxk\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:26.247141 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.247123    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb70c855-2622-4b0e-8229-58bc9085d0ed-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:26.366978 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.366951    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"]
Sep 16 01:13:26.373300 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.373279    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0ee1bf/sub-test-model-0ee1bf-kserve-579d7f7cd5-z8tq8"]
Sep 16 01:13:26.772790 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:26.772762    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb70c855-2622-4b0e-8229-58bc9085d0ed" path="/var/lib/kubelet/pods/eb70c855-2622-4b0e-8229-58bc9085d0ed/volumes"
Sep 16 01:13:30.764875 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:30.764842    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:13:35.022063 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:35.022033    2573 scope.go:117] "RemoveContainer" containerID="c8c6b33044ff96943c9436fa5fbe547acc1f9268cfaa96c3e999f44188cf9afc"
Sep 16 01:13:35.030587 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:35.030568    2573 scope.go:117] "RemoveContainer" containerID="f81a0993c84888d1289d1588b4fe7b15d2c43f6bfde3b9c42061ef2571d84dfc"
Sep 16 01:13:35.039631 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:35.039612    2573 scope.go:117] "RemoveContainer" containerID="aa0c835e989b6c573138dc21fd05d628383d4b85777132aea7b91477d1f6ffb0"
Sep 16 01:13:51.404982 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:51.404948    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"]
Sep 16 01:13:51.405443 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:51.405190    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy" containerID="cri-o://1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a" gracePeriod=30
Sep 16 01:13:56.667331 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.667275    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:13:56.741680 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.741642    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-certs\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.741840 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.741693    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-data\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.741840 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.741723    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-envoy\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.741840 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.741754    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d8m8\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-kube-api-access-5d8m8\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.741840 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.741794    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-podinfo\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.741840 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.741825    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-credential-socket\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.742111 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.741846    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-socket\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.742111 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.741921    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:56.742111 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742034    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istiod-ca-cert\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.742111 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742047    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:56.742111 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742071    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-token\") pod \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\" (UID: \"a2ea7b01-a491-46d7-bf42-35eb2bd41463\") "
Sep 16 01:13:56.742409 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742386    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:13:56.742543 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742503    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:56.742639 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742544    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:56.742639 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742558    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:56.742773 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742750    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:56.742967 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.742945    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:56.744579 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.744505    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-kube-api-access-5d8m8" (OuterVolumeSpecName: "kube-api-access-5d8m8") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "kube-api-access-5d8m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:13:56.744733 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.744707    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:13:56.745046 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.745023    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:13:56.745302 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.745268    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2ea7b01-a491-46d7-bf42-35eb2bd41463" (UID: "a2ea7b01-a491-46d7-bf42-35eb2bd41463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:13:56.844139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.844097    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:56.844234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.844141    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d8m8\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-kube-api-access-5d8m8\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:56.844234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.844156    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:56.844234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.844170    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:56.844234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.844184    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ea7b01-a491-46d7-bf42-35eb2bd41463-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:56.844234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:56.844196    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ea7b01-a491-46d7-bf42-35eb2bd41463-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:13:57.231388 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.231350    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerID="1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a" exitCode=0
Sep 16 01:13:57.231596 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.231397    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"
Sep 16 01:13:57.231596 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.231435    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" event={"ID":"a2ea7b01-a491-46d7-bf42-35eb2bd41463","Type":"ContainerDied","Data":"1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a"}
Sep 16 01:13:57.231596 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.231503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26" event={"ID":"a2ea7b01-a491-46d7-bf42-35eb2bd41463","Type":"ContainerDied","Data":"8e4429aedaf4bde8ad2c3d17f1d24f3fcf2aa728c5d439d2b3b3a4992369a397"}
Sep 16 01:13:57.231596 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.231561    2573 scope.go:117] "RemoveContainer" containerID="1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a"
Sep 16 01:13:57.242729 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.242696    2573 scope.go:117] "RemoveContainer" containerID="3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229"
Sep 16 01:13:57.248402 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.248383    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"]
Sep 16 01:13:57.251883 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.251862    2573 scope.go:117] "RemoveContainer" containerID="1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a"
Sep 16 01:13:57.252034 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.252016    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e7d6db19-openshift-default-6476c67568-mrn26"]
Sep 16 01:13:57.252141 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:13:57.252123    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a\": container with ID starting with 1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a not found: ID does not exist" containerID="1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a"
Sep 16 01:13:57.252182 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.252151    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a"} err="failed to get container status \"1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a\": rpc error: code = NotFound desc = could not find container \"1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a\": container with ID starting with 1d28e5726176206fba8532f2e23be7b920bef9df961f52175243e50f73e9be4a not found: ID does not exist"
Sep 16 01:13:57.252182 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.252168    2573 scope.go:117] "RemoveContainer" containerID="3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229"
Sep 16 01:13:57.252431 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:13:57.252414    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229\": container with ID starting with 3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229 not found: ID does not exist" containerID="3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229"
Sep 16 01:13:57.252487 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:57.252434    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229"} err="failed to get container status \"3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229\": rpc error: code = NotFound desc = could not find container \"3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229\": container with ID starting with 3793d03d63e7181ad718399ac1c7521fd5697717af7f9a848ce250d1b5cdd229 not found: ID does not exist"
Sep 16 01:13:58.771324 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:13:58.771287    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" path="/var/lib/kubelet/pods/a2ea7b01-a491-46d7-bf42-35eb2bd41463/volumes"
Sep 16 01:14:19.765142 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:19.765105    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:14:34.979501 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.979470    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"]
Sep 16 01:14:34.980190 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980158    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy"
Sep 16 01:14:34.980190 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980178    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy"
Sep 16 01:14:34.980321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980200    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47606ab-0865-48fe-a2f8-101237cdee84" containerName="main"
Sep 16 01:14:34.980321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980208    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47606ab-0865-48fe-a2f8-101237cdee84" containerName="main"
Sep 16 01:14:34.980321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980239    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5" containerName="payload-pre-processing"
Sep 16 01:14:34.980321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980248    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5" containerName="payload-pre-processing"
Sep 16 01:14:34.980321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980264    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb70c855-2622-4b0e-8229-58bc9085d0ed" containerName="main"
Sep 16 01:14:34.980321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980273    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb70c855-2622-4b0e-8229-58bc9085d0ed" containerName="main"
Sep 16 01:14:34.980321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980282    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66d961a-1482-48fc-bd97-c39e95b5d040" containerName="revoke-keys"
Sep 16 01:14:34.980321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980290    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66d961a-1482-48fc-bd97-c39e95b5d040" containerName="revoke-keys"
Sep 16 01:14:34.980739 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980388    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66d961a-1482-48fc-bd97-c39e95b5d040" containerName="revoke-keys"
Sep 16 01:14:34.980739 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980403    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1bb2c59-f4ff-4cce-b8f2-5080e8112fb5" containerName="payload-pre-processing"
Sep 16 01:14:34.980739 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980415    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb70c855-2622-4b0e-8229-58bc9085d0ed" containerName="main"
Sep 16 01:14:34.980739 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980425    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy"
Sep 16 01:14:34.980739 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.980434    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47606ab-0865-48fe-a2f8-101237cdee84" containerName="main"
Sep 16 01:14:34.983740 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.983719    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:34.985818 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.985800    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-089563-openshift-default-dockercfg-6tqqv\""
Sep 16 01:14:34.994321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:34.994297    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"]
Sep 16 01:14:35.104337 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104309    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-socket\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.104532 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104349    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-envoy\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.104532 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104470    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-token\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.104532 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104509    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-certs\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.104691 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104562    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-podinfo\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.104691 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-data\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.104768 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istiod-ca-cert\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.104814 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-credential-socket\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.104851 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.104818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l26s\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-kube-api-access-8l26s\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.205631 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istiod-ca-cert\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.205838 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-credential-socket\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.205838 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l26s\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-kube-api-access-8l26s\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.205838 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205766    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-socket\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.205838 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-envoy\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.206058 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-token\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.206058 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-certs\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.206058 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-podinfo\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.206058 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.205986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-data\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.206890 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.206453    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-data\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.206890 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.206590    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-socket\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.210992 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.207078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-credential-socket\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.210992 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.207314    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-certs\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.210992 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.208187    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istiod-ca-cert\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.210992 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.209898    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-envoy\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.211305 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.211179    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-podinfo\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.215791 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.215748    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-token\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.215887 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.215845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l26s\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-kube-api-access-8l26s\") pod \"e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") " pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.295379 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.295319    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:35.574750 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.574729    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"]
Sep 16 01:14:35.576826 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:14:35.576799    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3d51f7_e75a_4ed3_8496_0530c38e9776.slice/crio-c33c77615d00e6e4b791996fee1e8a210b140040f1968457dc499ddb7510a543 WatchSource:0}: Error finding container c33c77615d00e6e4b791996fee1e8a210b140040f1968457dc499ddb7510a543: Status 404 returned error can't find the container with id c33c77615d00e6e4b791996fee1e8a210b140040f1968457dc499ddb7510a543
Sep 16 01:14:35.579338 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.579302    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:14:35.579414 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.579387    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:14:35.579453 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:35.579435    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:14:36.462247 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:36.462203    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" event={"ID":"ba3d51f7-e75a-4ed3-8496-0530c38e9776","Type":"ContainerStarted","Data":"4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48"}
Sep 16 01:14:36.462692 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:36.462273    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" event={"ID":"ba3d51f7-e75a-4ed3-8496-0530c38e9776","Type":"ContainerStarted","Data":"c33c77615d00e6e4b791996fee1e8a210b140040f1968457dc499ddb7510a543"}
Sep 16 01:14:36.480913 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:36.480860    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" podStartSLOduration=2.480844697 podStartE2EDuration="2.480844697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:36.479236698 +0000 UTC m=+1862.182058891" watchObservedRunningTime="2026-09-16 01:14:36.480844697 +0000 UTC m=+1862.183666892"
Sep 16 01:14:37.296347 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:37.296311    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:37.301824 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:37.301802    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:37.468290 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:37.468259    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:37.469605 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:37.469584    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:14:42.065391 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.065356    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"]
Sep 16 01:14:42.065882 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.065808    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy"
Sep 16 01:14:42.065882 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.065820    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy"
Sep 16 01:14:42.065972 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.065912    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ea7b01-a491-46d7-bf42-35eb2bd41463" containerName="istio-proxy"
Sep 16 01:14:42.068886 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.068870    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:42.070920 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.070895    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-089563-dockercfg-7k9jb\""
Sep 16 01:14:42.071046 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.070897    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-089563\""
Sep 16 01:14:42.079968 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.079942    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"]
Sep 16 01:14:42.098514 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.098489    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"]
Sep 16 01:14:42.102133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.102115    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:42.112105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.112079    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"]
Sep 16 01:14:42.186547 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.185128    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd589a41-4148-491c-8441-c9ab7f546619-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq\" (UID: \"fd589a41-4148-491c-8441-c9ab7f546619\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:42.186547 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.185191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2g78\" (UniqueName: \"kubernetes.io/projected/fd589a41-4148-491c-8441-c9ab7f546619-kube-api-access-g2g78\") pod \"payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq\" (UID: \"fd589a41-4148-491c-8441-c9ab7f546619\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:42.286293 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.286258    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ceff0d-c76e-4065-ad4a-efb469675d40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2\" (UID: \"27ceff0d-c76e-4065-ad4a-efb469675d40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:42.286492 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.286347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd589a41-4148-491c-8441-c9ab7f546619-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq\" (UID: \"fd589a41-4148-491c-8441-c9ab7f546619\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:42.286492 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.286371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2g78\" (UniqueName: \"kubernetes.io/projected/fd589a41-4148-491c-8441-c9ab7f546619-kube-api-access-g2g78\") pod \"payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq\" (UID: \"fd589a41-4148-491c-8441-c9ab7f546619\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:42.286492 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.286387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h774c\" (UniqueName: \"kubernetes.io/projected/27ceff0d-c76e-4065-ad4a-efb469675d40-kube-api-access-h774c\") pod \"payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2\" (UID: \"27ceff0d-c76e-4065-ad4a-efb469675d40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:42.287077 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.287053    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd589a41-4148-491c-8441-c9ab7f546619-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq\" (UID: \"fd589a41-4148-491c-8441-c9ab7f546619\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:42.293579 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.293558    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2g78\" (UniqueName: \"kubernetes.io/projected/fd589a41-4148-491c-8441-c9ab7f546619-kube-api-access-g2g78\") pod \"payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq\" (UID: \"fd589a41-4148-491c-8441-c9ab7f546619\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:42.381398 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.381325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:42.386976 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.386951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h774c\" (UniqueName: \"kubernetes.io/projected/27ceff0d-c76e-4065-ad4a-efb469675d40-kube-api-access-h774c\") pod \"payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2\" (UID: \"27ceff0d-c76e-4065-ad4a-efb469675d40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:42.387091 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.387053    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ceff0d-c76e-4065-ad4a-efb469675d40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2\" (UID: \"27ceff0d-c76e-4065-ad4a-efb469675d40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:42.387733 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.387715    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ceff0d-c76e-4065-ad4a-efb469675d40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2\" (UID: \"27ceff0d-c76e-4065-ad4a-efb469675d40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:42.394804 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.394779    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h774c\" (UniqueName: \"kubernetes.io/projected/27ceff0d-c76e-4065-ad4a-efb469675d40-kube-api-access-h774c\") pod \"payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2\" (UID: \"27ceff0d-c76e-4065-ad4a-efb469675d40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:42.415172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.415144    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:42.658555 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.658512    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"]
Sep 16 01:14:42.660649 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:14:42.660628    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd589a41_4148_491c_8441_c9ab7f546619.slice/crio-ebf2b22862dd43bd53991c1c13b6fe708d80b91f7ea755329b255e0522708913 WatchSource:0}: Error finding container ebf2b22862dd43bd53991c1c13b6fe708d80b91f7ea755329b255e0522708913: Status 404 returned error can't find the container with id ebf2b22862dd43bd53991c1c13b6fe708d80b91f7ea755329b255e0522708913
Sep 16 01:14:42.702150 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:42.702124    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"]
Sep 16 01:14:42.712597 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:14:42.712570    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ceff0d_c76e_4065_ad4a_efb469675d40.slice/crio-7842db9ad1b1fcba1e079dfa967deeadbe7411c433e0009906e06c4b55d1ff7f WatchSource:0}: Error finding container 7842db9ad1b1fcba1e079dfa967deeadbe7411c433e0009906e06c4b55d1ff7f: Status 404 returned error can't find the container with id 7842db9ad1b1fcba1e079dfa967deeadbe7411c433e0009906e06c4b55d1ff7f
Sep 16 01:14:43.505436 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:43.505395    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2" event={"ID":"27ceff0d-c76e-4065-ad4a-efb469675d40","Type":"ContainerStarted","Data":"2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38"}
Sep 16 01:14:43.505917 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:43.505457    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2" event={"ID":"27ceff0d-c76e-4065-ad4a-efb469675d40","Type":"ContainerStarted","Data":"7842db9ad1b1fcba1e079dfa967deeadbe7411c433e0009906e06c4b55d1ff7f"}
Sep 16 01:14:43.505917 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:43.505544    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:43.507134 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:43.507107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq" event={"ID":"fd589a41-4148-491c-8441-c9ab7f546619","Type":"ContainerStarted","Data":"57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf"}
Sep 16 01:14:43.507270 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:43.507146    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq" event={"ID":"fd589a41-4148-491c-8441-c9ab7f546619","Type":"ContainerStarted","Data":"ebf2b22862dd43bd53991c1c13b6fe708d80b91f7ea755329b255e0522708913"}
Sep 16 01:14:43.507270 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:43.507217    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:43.523314 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:43.523269    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2" podStartSLOduration=1.523257288 podStartE2EDuration="1.523257288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:43.520423972 +0000 UTC m=+1869.223246192" watchObservedRunningTime="2026-09-16 01:14:43.523257288 +0000 UTC m=+1869.226079473"
Sep 16 01:14:43.535345 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:43.535303    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq" podStartSLOduration=1.535288726 podStartE2EDuration="1.535288726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:43.533536433 +0000 UTC m=+1869.236358626" watchObservedRunningTime="2026-09-16 01:14:43.535288726 +0000 UTC m=+1869.238110920"
Sep 16 01:14:52.873644 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:52.873610    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"]
Sep 16 01:14:52.886615 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:52.886584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:52.888823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:52.888796    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"]
Sep 16 01:14:52.889008 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:52.888819    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ad13e5\""
Sep 16 01:14:52.999437 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:52.999403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg\" (UID: \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:52.999823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:52.999804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnmz\" (UniqueName: \"kubernetes.io/projected/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-kube-api-access-chnmz\") pod \"payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg\" (UID: \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:53.101452 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:53.100712    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnmz\" (UniqueName: \"kubernetes.io/projected/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-kube-api-access-chnmz\") pod \"payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg\" (UID: \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:53.101452 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:53.100799    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg\" (UID: \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:53.101452 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:53.101415    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg\" (UID: \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:53.107502 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:53.107478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnmz\" (UniqueName: \"kubernetes.io/projected/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-kube-api-access-chnmz\") pod \"payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg\" (UID: \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:53.199312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:53.199278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:53.511606 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:53.511575    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"]
Sep 16 01:14:53.513922 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:14:53.513884    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423e27a8_c53a_47f3_9728_f9f41c9c4a9d.slice/crio-cdec2a8f7f3d7182b345837a6c2105f5cbddc1298331cff016bb446e9f9a67cd WatchSource:0}: Error finding container cdec2a8f7f3d7182b345837a6c2105f5cbddc1298331cff016bb446e9f9a67cd: Status 404 returned error can't find the container with id cdec2a8f7f3d7182b345837a6c2105f5cbddc1298331cff016bb446e9f9a67cd
Sep 16 01:14:53.572496 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:53.572465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg" event={"ID":"423e27a8-c53a-47f3-9728-f9f41c9c4a9d","Type":"ContainerStarted","Data":"cdec2a8f7f3d7182b345837a6c2105f5cbddc1298331cff016bb446e9f9a67cd"}
Sep 16 01:14:54.514969 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:54.514930    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:14:54.515347 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:54.515139    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:14:54.582349 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:54.582311    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg" event={"ID":"423e27a8-c53a-47f3-9728-f9f41c9c4a9d","Type":"ContainerStarted","Data":"9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07"}
Sep 16 01:14:54.582762 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:54.582745    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:14:54.599941 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:54.599898    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg" podStartSLOduration=2.599884443 podStartE2EDuration="2.599884443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:54.597722153 +0000 UTC m=+1880.300544349" watchObservedRunningTime="2026-09-16 01:14:54.599884443 +0000 UTC m=+1880.302706637"
Sep 16 01:14:56.765253 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:56.765214    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:14:59.382495 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.382460    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"]
Sep 16 01:14:59.385400 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.385384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.387662 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.387642    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-195f51-openshift-default-dockercfg-zwm7m\""
Sep 16 01:14:59.396479 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.396456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"]
Sep 16 01:14:59.470829 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.470797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-podinfo\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.471166 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.470969    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-certs\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.471166 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.471051    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-socket\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.471166 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.471081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istiod-ca-cert\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.471166 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.471126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-token\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.471479 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.471183    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-envoy\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.471479 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.471264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxbw\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-kube-api-access-dhxbw\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.471479 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.471282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-data\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.471479 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.471341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-credential-socket\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.572516 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572485    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-token\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.572705 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-envoy\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.572705 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxbw\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-kube-api-access-dhxbw\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.572705 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-data\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.572849 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572707    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-credential-socket\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.572849 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572760    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-podinfo\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.572849 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-certs\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.573059 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572871    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-socket\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.573059 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.572898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istiod-ca-cert\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.573187 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.573082    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-data\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.573248 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.573202    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-socket\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.573336 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.573314    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-certs\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.573392 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.573351    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-credential-socket\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.573564 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.573548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istiod-ca-cert\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.575061 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.575040    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-envoy\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.575331 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.575315    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-podinfo\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.580709 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.580691    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-token\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.580893 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.580875    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxbw\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-kube-api-access-dhxbw\") pod \"e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") " pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.612949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.612929    2573 generic.go:356] "Generic (PLEG): container finished" podID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerID="4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48" exitCode=0
Sep 16 01:14:59.613042 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.612962    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" event={"ID":"ba3d51f7-e75a-4ed3-8496-0530c38e9776","Type":"ContainerDied","Data":"4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48"}
Sep 16 01:14:59.613331 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.613309    2573 scope.go:117] "RemoveContainer" containerID="4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48"
Sep 16 01:14:59.614370 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.614344    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:14:59.614430 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.614413    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:14:59.614474 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.614446    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:14:59.697368 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.697336    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:14:59.987028 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.987001    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"]
Sep 16 01:14:59.989252 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:14:59.989153    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce110ad_d891_4dfa_a4ac_b0fe85c9cdd7.slice/crio-f4fe1118dc3b465980f062574bc121469ad44332ebc81621fe072f7ec99c48e7 WatchSource:0}: Error finding container f4fe1118dc3b465980f062574bc121469ad44332ebc81621fe072f7ec99c48e7: Status 404 returned error can't find the container with id f4fe1118dc3b465980f062574bc121469ad44332ebc81621fe072f7ec99c48e7
Sep 16 01:14:59.991862 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.991830    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:14:59.991955 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.991894    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:14:59.991955 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:14:59.991923    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:00.136238 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.136181    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"]
Sep 16 01:15:00.139764 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.139738    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq"]
Sep 16 01:15:00.139908 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.139862    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"
Sep 16 01:15:00.142045 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.141914    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:15:00.143011 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.142988    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"]
Sep 16 01:15:00.143335 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.143080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq"
Sep 16 01:15:00.148419 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.148401    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"]
Sep 16 01:15:00.148962 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.148941    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"
Sep 16 01:15:00.151962 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.151937    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"]
Sep 16 01:15:00.152172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.152135    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq"]
Sep 16 01:15:00.152333 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.152009    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"
Sep 16 01:15:00.155546 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.155503    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"]
Sep 16 01:15:00.168007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.160975    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"
Sep 16 01:15:00.168007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.161191    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"]
Sep 16 01:15:00.173073 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.173050    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq"
Sep 16 01:15:00.225718 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.225688    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"
Sep 16 01:15:00.240589 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.240425    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"
Sep 16 01:15:00.493679 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.493650    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"]
Sep 16 01:15:00.495231 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:15:00.495182    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabd2813_0a40_410b_8ac5_8e794bd4ee9c.slice/crio-2399147e648a5aac7c3602c96ccb654a41075b81e95d7d56b5cf0f31933fb322 WatchSource:0}: Error finding container 2399147e648a5aac7c3602c96ccb654a41075b81e95d7d56b5cf0f31933fb322: Status 404 returned error can't find the container with id 2399147e648a5aac7c3602c96ccb654a41075b81e95d7d56b5cf0f31933fb322
Sep 16 01:15:00.511384 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.511364    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq"]
Sep 16 01:15:00.512684 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:15:00.512662    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ddcd7a_df8e_434a_82f9_c93022c7beb5.slice/crio-50488e4d29c033a7d6d44bea8b9210b4a4000fdc62be293fa2d95c8b4b361a9a WatchSource:0}: Error finding container 50488e4d29c033a7d6d44bea8b9210b4a4000fdc62be293fa2d95c8b4b361a9a: Status 404 returned error can't find the container with id 50488e4d29c033a7d6d44bea8b9210b4a4000fdc62be293fa2d95c8b4b361a9a
Sep 16 01:15:00.555780 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.555753    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"]
Sep 16 01:15:00.560355 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:15:00.560327    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4ee8b4_0c31_4f9e_9768_32c486811489.slice/crio-ea72add55714a1ee6c4592e3a4a2cc94184e7ea6ecdef70d6e4e7c8a53e8c326 WatchSource:0}: Error finding container ea72add55714a1ee6c4592e3a4a2cc94184e7ea6ecdef70d6e4e7c8a53e8c326: Status 404 returned error can't find the container with id ea72add55714a1ee6c4592e3a4a2cc94184e7ea6ecdef70d6e4e7c8a53e8c326
Sep 16 01:15:00.561313 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.561191    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"]
Sep 16 01:15:00.561935 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:15:00.561907    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16555e0_f6c2_43b1_9be4_21ad80cfa30d.slice/crio-a3ef61eb5f743c7be795ba6a092b3a9fdd2f95c0a7ae910a200e9d33866d039e WatchSource:0}: Error finding container a3ef61eb5f743c7be795ba6a092b3a9fdd2f95c0a7ae910a200e9d33866d039e: Status 404 returned error can't find the container with id a3ef61eb5f743c7be795ba6a092b3a9fdd2f95c0a7ae910a200e9d33866d039e
Sep 16 01:15:00.621594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.621566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" event={"ID":"ba3d51f7-e75a-4ed3-8496-0530c38e9776","Type":"ContainerStarted","Data":"5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3"}
Sep 16 01:15:00.622941 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.622907    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l" event={"ID":"ff4ee8b4-0c31-4f9e-9768-32c486811489","Type":"ContainerStarted","Data":"ea72add55714a1ee6c4592e3a4a2cc94184e7ea6ecdef70d6e4e7c8a53e8c326"}
Sep 16 01:15:00.624248 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.624228    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt" event={"ID":"cabd2813-0a40-410b-8ac5-8e794bd4ee9c","Type":"ContainerStarted","Data":"2399147e648a5aac7c3602c96ccb654a41075b81e95d7d56b5cf0f31933fb322"}
Sep 16 01:15:00.625553 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.625510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq" event={"ID":"12ddcd7a-df8e-434a-82f9-c93022c7beb5","Type":"ContainerStarted","Data":"50488e4d29c033a7d6d44bea8b9210b4a4000fdc62be293fa2d95c8b4b361a9a"}
Sep 16 01:15:00.627042 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.627016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" event={"ID":"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7","Type":"ContainerStarted","Data":"fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34"}
Sep 16 01:15:00.627129 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.627062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" event={"ID":"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7","Type":"ContainerStarted","Data":"f4fe1118dc3b465980f062574bc121469ad44332ebc81621fe072f7ec99c48e7"}
Sep 16 01:15:00.628320 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.628295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8" event={"ID":"e16555e0-f6c2-43b1-9be4-21ad80cfa30d","Type":"ContainerStarted","Data":"a3ef61eb5f743c7be795ba6a092b3a9fdd2f95c0a7ae910a200e9d33866d039e"}
Sep 16 01:15:00.671112 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.671048    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" podStartSLOduration=1.6710304790000001 podStartE2EDuration="1.671030479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:00.668634259 +0000 UTC m=+1886.371456446" watchObservedRunningTime="2026-09-16 01:15:00.671030479 +0000 UTC m=+1886.373852674"
Sep 16 01:15:00.697768 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:00.697741    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:15:01.296626 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.296507    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:15:01.297139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.297108    2573 patch_prober.go:28] interesting pod/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 16 01:15:01.297277 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.297161    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 01:15:01.639828 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.639728    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt" event={"ID":"cabd2813-0a40-410b-8ac5-8e794bd4ee9c","Type":"ContainerStarted","Data":"eb2143100ba2e89308cdf55fe9932b85cc38c41e60739af0077cd6fd410c23b7"}
Sep 16 01:15:01.643346 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.643313    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq" event={"ID":"12ddcd7a-df8e-434a-82f9-c93022c7beb5","Type":"ContainerStarted","Data":"099bde4752709b9cb29a2c5d4ac7af1e44d9a08640fb7c9e62ea3e349730d4d8"}
Sep 16 01:15:01.650558 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.650497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8" event={"ID":"e16555e0-f6c2-43b1-9be4-21ad80cfa30d","Type":"ContainerStarted","Data":"2b5faf5c764aa91c24a69841cf7745e2fb6b14d6fa8abdcba4b6699f9fd9ec0d"}
Sep 16 01:15:01.653515 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.653483    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l" event={"ID":"ff4ee8b4-0c31-4f9e-9768-32c486811489","Type":"ContainerStarted","Data":"d5fc4208e812ae5b2417301134a466e0fb165df1348916091e0ee42f87589d1d"}
Sep 16 01:15:01.660408 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.659621    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt" podStartSLOduration=1.659601521 podStartE2EDuration="1.659601521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:01.655792701 +0000 UTC m=+1887.358614897" watchObservedRunningTime="2026-09-16 01:15:01.659601521 +0000 UTC m=+1887.362423716"
Sep 16 01:15:01.672089 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.672025    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l" podStartSLOduration=1.672009174 podStartE2EDuration="1.672009174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:01.670424361 +0000 UTC m=+1887.373246556" watchObservedRunningTime="2026-09-16 01:15:01.672009174 +0000 UTC m=+1887.374831368"
Sep 16 01:15:01.687886 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.687831    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8" podStartSLOduration=1.687814033 podStartE2EDuration="1.687814033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:01.686366044 +0000 UTC m=+1887.389188240" watchObservedRunningTime="2026-09-16 01:15:01.687814033 +0000 UTC m=+1887.390636228"
Sep 16 01:15:01.701785 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.701730    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq" podStartSLOduration=1.7017108429999999 podStartE2EDuration="1.701710843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:01.699359394 +0000 UTC m=+1887.402181589" watchObservedRunningTime="2026-09-16 01:15:01.701710843 +0000 UTC m=+1887.404533037"
Sep 16 01:15:01.703713 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:01.703687    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:15:02.296321 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.296289    2573 patch_prober.go:28] interesting pod/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 16 01:15:02.296596 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.296574    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 01:15:02.663437 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.663337    2573 generic.go:356] "Generic (PLEG): container finished" podID="ff4ee8b4-0c31-4f9e-9768-32c486811489" containerID="d5fc4208e812ae5b2417301134a466e0fb165df1348916091e0ee42f87589d1d" exitCode=0
Sep 16 01:15:02.663437 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.663418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l" event={"ID":"ff4ee8b4-0c31-4f9e-9768-32c486811489","Type":"ContainerDied","Data":"d5fc4208e812ae5b2417301134a466e0fb165df1348916091e0ee42f87589d1d"}
Sep 16 01:15:02.665222 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.665199    2573 generic.go:356] "Generic (PLEG): container finished" podID="cabd2813-0a40-410b-8ac5-8e794bd4ee9c" containerID="eb2143100ba2e89308cdf55fe9932b85cc38c41e60739af0077cd6fd410c23b7" exitCode=0
Sep 16 01:15:02.665352 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.665277    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt" event={"ID":"cabd2813-0a40-410b-8ac5-8e794bd4ee9c","Type":"ContainerDied","Data":"eb2143100ba2e89308cdf55fe9932b85cc38c41e60739af0077cd6fd410c23b7"}
Sep 16 01:15:02.667064 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.667040    2573 generic.go:356] "Generic (PLEG): container finished" podID="12ddcd7a-df8e-434a-82f9-c93022c7beb5" containerID="099bde4752709b9cb29a2c5d4ac7af1e44d9a08640fb7c9e62ea3e349730d4d8" exitCode=0
Sep 16 01:15:02.667201 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.667167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq" event={"ID":"12ddcd7a-df8e-434a-82f9-c93022c7beb5","Type":"ContainerDied","Data":"099bde4752709b9cb29a2c5d4ac7af1e44d9a08640fb7c9e62ea3e349730d4d8"}
Sep 16 01:15:02.668673 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.668652    2573 generic.go:356] "Generic (PLEG): container finished" podID="e16555e0-f6c2-43b1-9be4-21ad80cfa30d" containerID="2b5faf5c764aa91c24a69841cf7745e2fb6b14d6fa8abdcba4b6699f9fd9ec0d" exitCode=0
Sep 16 01:15:02.668778 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.668705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8" event={"ID":"e16555e0-f6c2-43b1-9be4-21ad80cfa30d","Type":"ContainerDied","Data":"2b5faf5c764aa91c24a69841cf7745e2fb6b14d6fa8abdcba4b6699f9fd9ec0d"}
Sep 16 01:15:02.668945 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.668922    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:15:02.669911 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:02.669894    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:15:03.296775 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:03.296732    2573 patch_prober.go:28] interesting pod/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 16 01:15:03.296967 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:03.296804    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 01:15:03.904814 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:03.904699    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"
Sep 16 01:15:03.975442 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:03.975414    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"
Sep 16 01:15:03.987127 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:03.987095    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"
Sep 16 01:15:03.992463 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:03.992438    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq"
Sep 16 01:15:04.296168 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.296134    2573 patch_prober.go:28] interesting pod/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 16 01:15:04.296342 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.296189    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 01:15:04.682181 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.682091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8" event={"ID":"e16555e0-f6c2-43b1-9be4-21ad80cfa30d","Type":"ContainerDied","Data":"a3ef61eb5f743c7be795ba6a092b3a9fdd2f95c0a7ae910a200e9d33866d039e"}
Sep 16 01:15:04.682181 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.682176    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ef61eb5f743c7be795ba6a092b3a9fdd2f95c0a7ae910a200e9d33866d039e"
Sep 16 01:15:04.682409 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.682243    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"
Sep 16 01:15:04.684348 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.684316    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"
Sep 16 01:15:04.684770 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.684326    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l" event={"ID":"ff4ee8b4-0c31-4f9e-9768-32c486811489","Type":"ContainerDied","Data":"ea72add55714a1ee6c4592e3a4a2cc94184e7ea6ecdef70d6e4e7c8a53e8c326"}
Sep 16 01:15:04.684877 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.684822    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea72add55714a1ee6c4592e3a4a2cc94184e7ea6ecdef70d6e4e7c8a53e8c326"
Sep 16 01:15:04.686130 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.685784    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"
Sep 16 01:15:04.686130 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.685803    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt" event={"ID":"cabd2813-0a40-410b-8ac5-8e794bd4ee9c","Type":"ContainerDied","Data":"2399147e648a5aac7c3602c96ccb654a41075b81e95d7d56b5cf0f31933fb322"}
Sep 16 01:15:04.686130 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.685840    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2399147e648a5aac7c3602c96ccb654a41075b81e95d7d56b5cf0f31933fb322"
Sep 16 01:15:04.688020 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.687992    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq"
Sep 16 01:15:04.688135 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.688033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-c97vq" event={"ID":"12ddcd7a-df8e-434a-82f9-c93022c7beb5","Type":"ContainerDied","Data":"50488e4d29c033a7d6d44bea8b9210b4a4000fdc62be293fa2d95c8b4b361a9a"}
Sep 16 01:15:04.688135 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:04.688068    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50488e4d29c033a7d6d44bea8b9210b4a4000fdc62be293fa2d95c8b4b361a9a"
Sep 16 01:15:05.296416 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:05.296376    2573 patch_prober.go:28] interesting pod/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 16 01:15:05.296846 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:05.296434    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 01:15:05.590379 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:05.590293    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:15:06.347664 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:06.347616    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:15:06.348270 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:06.348247    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:15:06.349229 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:06.349210    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:15:09.106843 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.106812    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"]
Sep 16 01:15:09.107325 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107312    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16555e0-f6c2-43b1-9be4-21ad80cfa30d" containerName="cleanup"
Sep 16 01:15:09.107365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107328    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16555e0-f6c2-43b1-9be4-21ad80cfa30d" containerName="cleanup"
Sep 16 01:15:09.107365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107343    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabd2813-0a40-410b-8ac5-8e794bd4ee9c" containerName="cleanup"
Sep 16 01:15:09.107365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107348    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabd2813-0a40-410b-8ac5-8e794bd4ee9c" containerName="cleanup"
Sep 16 01:15:09.107365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107361    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ddcd7a-df8e-434a-82f9-c93022c7beb5" containerName="cleanup"
Sep 16 01:15:09.107542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107366    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ddcd7a-df8e-434a-82f9-c93022c7beb5" containerName="cleanup"
Sep 16 01:15:09.107542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107380    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff4ee8b4-0c31-4f9e-9768-32c486811489" containerName="cleanup"
Sep 16 01:15:09.107542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107385    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff4ee8b4-0c31-4f9e-9768-32c486811489" containerName="cleanup"
Sep 16 01:15:09.107542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107456    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff4ee8b4-0c31-4f9e-9768-32c486811489" containerName="cleanup"
Sep 16 01:15:09.107542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107464    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ddcd7a-df8e-434a-82f9-c93022c7beb5" containerName="cleanup"
Sep 16 01:15:09.107542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107470    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16555e0-f6c2-43b1-9be4-21ad80cfa30d" containerName="cleanup"
Sep 16 01:15:09.107542 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.107478    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cabd2813-0a40-410b-8ac5-8e794bd4ee9c" containerName="cleanup"
Sep 16 01:15:09.109850 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.109833    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.112155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.112135    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-00f5c9-openshift-default-dockercfg-ph5gw\""
Sep 16 01:15:09.123794 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.123768    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"]
Sep 16 01:15:09.206079 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.206041    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-credential-socket\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.207692 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.207666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-data\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.207917 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.207899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-certs\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.208397 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.208284    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-podinfo\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.208698 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.208678    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-envoy\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.208872 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.208849    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e99831-2a48-46ae-aa89-a42ae583dee1-istiod-ca-cert\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.209073 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.209056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-token\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.209274 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.209257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krdrv\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-kube-api-access-krdrv\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.209470 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.209434    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-socket\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.310687 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.310655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-data\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.310882 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.310693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-certs\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.310882 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.310721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-podinfo\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.310882 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.310837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-envoy\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.310882 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.310867    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e99831-2a48-46ae-aa89-a42ae583dee1-istiod-ca-cert\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.310904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-token\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.310955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdrv\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-kube-api-access-krdrv\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.310996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-socket\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.311057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-credential-socket\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.311105    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-data\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.311141    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-certs\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311503 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.311389    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-credential-socket\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311503 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.311424    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-socket\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.311735 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.311712    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e99831-2a48-46ae-aa89-a42ae583dee1-istiod-ca-cert\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.313266 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.313250    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-envoy\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.313341 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.313326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-podinfo\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.318362 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.318336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-token\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.318791 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.318770    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krdrv\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-kube-api-access-krdrv\") pod \"e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") " pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.425901 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.425825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:09.703227 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.703202    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"]
Sep 16 01:15:09.704513 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:15:09.704489    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e99831_2a48_46ae_aa89_a42ae583dee1.slice/crio-6bd3fabd80d6feccb18dabb7f273fe9574645d5c1cb3bec36895b2f4263fdefc WatchSource:0}: Error finding container 6bd3fabd80d6feccb18dabb7f273fe9574645d5c1cb3bec36895b2f4263fdefc: Status 404 returned error can't find the container with id 6bd3fabd80d6feccb18dabb7f273fe9574645d5c1cb3bec36895b2f4263fdefc
Sep 16 01:15:09.706851 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.706820    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:09.706947 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.706886    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:09.706947 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.706913    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:09.724328 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:09.724299    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" event={"ID":"17e99831-2a48-46ae-aa89-a42ae583dee1","Type":"ContainerStarted","Data":"6bd3fabd80d6feccb18dabb7f273fe9574645d5c1cb3bec36895b2f4263fdefc"}
Sep 16 01:15:10.733845 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:10.733796    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" event={"ID":"17e99831-2a48-46ae-aa89-a42ae583dee1","Type":"ContainerStarted","Data":"bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0"}
Sep 16 01:15:10.753298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:10.753244    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" podStartSLOduration=1.753228228 podStartE2EDuration="1.753228228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:10.750999916 +0000 UTC m=+1896.453822123" watchObservedRunningTime="2026-09-16 01:15:10.753228228 +0000 UTC m=+1896.456050481"
Sep 16 01:15:11.426819 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:11.426788    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:11.433442 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:11.433421    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:11.740939 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:11.740911    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:11.741918 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:11.741899    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:18.017375 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.017342    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"]
Sep 16 01:15:18.020373 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.020353    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.022432 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.022402    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-00f5c9\""
Sep 16 01:15:18.022585 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.022566    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-00f5c9-dockercfg-2ck77\""
Sep 16 01:15:18.037391 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.037360    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"]
Sep 16 01:15:18.103437 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.103407    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2207b6-7855-4c0b-83ca-3351ff0aa073-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c\" (UID: \"da2207b6-7855-4c0b-83ca-3351ff0aa073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.103652 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.103463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pjb\" (UniqueName: \"kubernetes.io/projected/da2207b6-7855-4c0b-83ca-3351ff0aa073-kube-api-access-57pjb\") pod \"payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c\" (UID: \"da2207b6-7855-4c0b-83ca-3351ff0aa073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.204106 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.204073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pjb\" (UniqueName: \"kubernetes.io/projected/da2207b6-7855-4c0b-83ca-3351ff0aa073-kube-api-access-57pjb\") pod \"payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c\" (UID: \"da2207b6-7855-4c0b-83ca-3351ff0aa073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.204312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.204252    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2207b6-7855-4c0b-83ca-3351ff0aa073-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c\" (UID: \"da2207b6-7855-4c0b-83ca-3351ff0aa073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.205003 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.204984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2207b6-7855-4c0b-83ca-3351ff0aa073-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c\" (UID: \"da2207b6-7855-4c0b-83ca-3351ff0aa073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.211432 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.211413    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pjb\" (UniqueName: \"kubernetes.io/projected/da2207b6-7855-4c0b-83ca-3351ff0aa073-kube-api-access-57pjb\") pod \"payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c\" (UID: \"da2207b6-7855-4c0b-83ca-3351ff0aa073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.352438 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.352348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.642000 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.641968    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"]
Sep 16 01:15:18.795346 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.795305    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c" event={"ID":"da2207b6-7855-4c0b-83ca-3351ff0aa073","Type":"ContainerStarted","Data":"c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75"}
Sep 16 01:15:18.795509 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.795368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c" event={"ID":"da2207b6-7855-4c0b-83ca-3351ff0aa073","Type":"ContainerStarted","Data":"c068e99b54a34ea018d2d30d4ecb624d53afdf1ad0d756966265c47d1d8f43f3"}
Sep 16 01:15:18.795509 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.795502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:18.812953 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:18.812908    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c" podStartSLOduration=0.812894108 podStartE2EDuration="812.894108ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:18.81004648 +0000 UTC m=+1904.512868669" watchObservedRunningTime="2026-09-16 01:15:18.812894108 +0000 UTC m=+1904.515716294"
Sep 16 01:15:24.844594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:24.844563    2573 generic.go:356] "Generic (PLEG): container finished" podID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerID="fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34" exitCode=0
Sep 16 01:15:24.844594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:24.844598    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" event={"ID":"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7","Type":"ContainerDied","Data":"fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34"}
Sep 16 01:15:24.845065 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:24.844960    2573 scope.go:117] "RemoveContainer" containerID="fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34"
Sep 16 01:15:24.846061 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:24.846035    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:24.846117 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:24.846101    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:24.846159 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:24.846135    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:25.863551 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:25.863491    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" event={"ID":"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7","Type":"ContainerStarted","Data":"d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1"}
Sep 16 01:15:26.698597 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:26.698555    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:15:26.698778 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:26.698755    2573 patch_prober.go:28] interesting pod/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 16 01:15:26.698832 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:26.698797    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 01:15:27.698378 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:27.698337    2573 patch_prober.go:28] interesting pod/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 16 01:15:27.698789 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:27.698412    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 01:15:28.698887 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:28.698846    2573 patch_prober.go:28] interesting pod/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 16 01:15:28.699398 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:28.698918    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 01:15:29.698287 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:29.698241    2573 patch_prober.go:28] interesting pod/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 16 01:15:29.698649 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:29.698303    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 01:15:29.804479 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:29.804449    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:15:30.698039 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:30.698004    2573 patch_prober.go:28] interesting pod/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 16 01:15:30.698220 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:30.698060    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 16 01:15:31.702716 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:31.702686    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:15:31.703098 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:31.702962    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:15:31.703657 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:31.703640    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:15:35.794257 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:35.794212    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"]
Sep 16 01:15:35.798000 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:35.797975    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"
Sep 16 01:15:35.800403 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:35.800380    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:15:35.800555 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:35.800434    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:15:35.805978 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:35.805954    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"]
Sep 16 01:15:35.903429 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:35.903391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5095b8fa-00d4-4e60-9931-ad9f51da67dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad13e5-229b9\" (UID: \"5095b8fa-00d4-4e60-9931-ad9f51da67dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"
Sep 16 01:15:36.005598 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:36.005556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5095b8fa-00d4-4e60-9931-ad9f51da67dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad13e5-229b9\" (UID: \"5095b8fa-00d4-4e60-9931-ad9f51da67dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"
Sep 16 01:15:36.006270 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:36.006246    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5095b8fa-00d4-4e60-9931-ad9f51da67dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad13e5-229b9\" (UID: \"5095b8fa-00d4-4e60-9931-ad9f51da67dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"
Sep 16 01:15:36.111185 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:36.111112    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"
Sep 16 01:15:36.386578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:36.386549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"]
Sep 16 01:15:36.947608 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:36.947575    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9" event={"ID":"5095b8fa-00d4-4e60-9931-ad9f51da67dc","Type":"ContainerStarted","Data":"7e8d62afe8cd51300a8d1847f7fd73df372c49464bafe09703eb8774e8e2d8e4"}
Sep 16 01:15:36.947999 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:36.947627    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9" event={"ID":"5095b8fa-00d4-4e60-9931-ad9f51da67dc","Type":"ContainerStarted","Data":"e3dd0c72ceb7a5ad234006efc5d0dc873e469017dc0d3adb17d6bfcd8577c31c"}
Sep 16 01:15:36.962229 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:36.962185    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9" podStartSLOduration=1.962170841 podStartE2EDuration="1.962170841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:36.96029726 +0000 UTC m=+1922.663119467" watchObservedRunningTime="2026-09-16 01:15:36.962170841 +0000 UTC m=+1922.664993074"
Sep 16 01:15:37.765031 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.765002    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:15:37.955071 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.955036    2573 generic.go:356] "Generic (PLEG): container finished" podID="5095b8fa-00d4-4e60-9931-ad9f51da67dc" containerID="7e8d62afe8cd51300a8d1847f7fd73df372c49464bafe09703eb8774e8e2d8e4" exitCode=0
Sep 16 01:15:37.955496 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.955124    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9" event={"ID":"5095b8fa-00d4-4e60-9931-ad9f51da67dc","Type":"ContainerDied","Data":"7e8d62afe8cd51300a8d1847f7fd73df372c49464bafe09703eb8774e8e2d8e4"}
Sep 16 01:15:37.956770 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.956749    2573 generic.go:356] "Generic (PLEG): container finished" podID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerID="bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0" exitCode=0
Sep 16 01:15:37.956860 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.956829    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" event={"ID":"17e99831-2a48-46ae-aa89-a42ae583dee1","Type":"ContainerDied","Data":"bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0"}
Sep 16 01:15:37.957215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.957200    2573 scope.go:117] "RemoveContainer" containerID="bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0"
Sep 16 01:15:37.958413 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.958387    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:37.958492 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.958452    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:37.958551 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:37.958507    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:15:38.125271 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:38.125242    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-87r96_b3056300-83f7-42c9-97ac-82fff85a22b8/payload-processing/0.log"
Sep 16 01:15:38.245438 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:38.245385    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-87r96_b3056300-83f7-42c9-97ac-82fff85a22b8/payload-processing/0.log"
Sep 16 01:15:38.971363 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:38.971315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" event={"ID":"17e99831-2a48-46ae-aa89-a42ae583dee1","Type":"ContainerStarted","Data":"d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745"}
Sep 16 01:15:39.166583 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.166543    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"
Sep 16 01:15:39.268300 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.268209    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5095b8fa-00d4-4e60-9931-ad9f51da67dc-maas-api-service-ca\") pod \"5095b8fa-00d4-4e60-9931-ad9f51da67dc\" (UID: \"5095b8fa-00d4-4e60-9931-ad9f51da67dc\") "
Sep 16 01:15:39.268600 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.268573    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5095b8fa-00d4-4e60-9931-ad9f51da67dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5095b8fa-00d4-4e60-9931-ad9f51da67dc" (UID: "5095b8fa-00d4-4e60-9931-ad9f51da67dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:15:39.369843 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.369803    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5095b8fa-00d4-4e60-9931-ad9f51da67dc-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:15:39.426604 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.426569    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:39.428126 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.428097    2573 patch_prober.go:28] interesting pod/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 16 01:15:39.428353 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.428326    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 16 01:15:39.979405 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.979374    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"
Sep 16 01:15:39.979925 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.979410    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9" event={"ID":"5095b8fa-00d4-4e60-9931-ad9f51da67dc","Type":"ContainerDied","Data":"e3dd0c72ceb7a5ad234006efc5d0dc873e469017dc0d3adb17d6bfcd8577c31c"}
Sep 16 01:15:39.979925 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:39.979466    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3dd0c72ceb7a5ad234006efc5d0dc873e469017dc0d3adb17d6bfcd8577c31c"
Sep 16 01:15:40.426251 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:40.426158    2573 patch_prober.go:28] interesting pod/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 16 01:15:40.426251 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:40.426227    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 16 01:15:41.426302 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:41.426270    2573 patch_prober.go:28] interesting pod/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 16 01:15:41.426736 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:41.426324    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 16 01:15:41.741312 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:41.741282    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:42.427061 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:42.427030    2573 patch_prober.go:28] interesting pod/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 16 01:15:42.427460 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:42.427093    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 16 01:15:43.426471 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:43.426432    2573 patch_prober.go:28] interesting pod/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 16 01:15:43.426685 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:43.426487    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 16 01:15:44.431810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:44.431772    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:44.432995 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:44.432971    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:15:51.588345 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:51.588309    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"]
Sep 16 01:15:51.589057 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:51.589026    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg" podUID="423e27a8-c53a-47f3-9728-f9f41c9c4a9d" containerName="payload-pre-processing" containerID="cri-o://9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07" gracePeriod=30
Sep 16 01:15:51.740268 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:51.740236    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"]
Sep 16 01:15:51.744150 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:51.744120    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad13e5-229b9"]
Sep 16 01:15:51.873693 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:51.873667    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:15:51.998680 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:51.998650    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-plugins-config-volume\") pod \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\" (UID: \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\") "
Sep 16 01:15:51.998892 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:51.998729    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chnmz\" (UniqueName: \"kubernetes.io/projected/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-kube-api-access-chnmz\") pod \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\" (UID: \"423e27a8-c53a-47f3-9728-f9f41c9c4a9d\") "
Sep 16 01:15:51.999076 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:51.999053    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "423e27a8-c53a-47f3-9728-f9f41c9c4a9d" (UID: "423e27a8-c53a-47f3-9728-f9f41c9c4a9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:15:52.000960 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.000936    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-kube-api-access-chnmz" (OuterVolumeSpecName: "kube-api-access-chnmz") pod "423e27a8-c53a-47f3-9728-f9f41c9c4a9d" (UID: "423e27a8-c53a-47f3-9728-f9f41c9c4a9d"). InnerVolumeSpecName "kube-api-access-chnmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:15:52.052264 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.052236    2573 generic.go:356] "Generic (PLEG): container finished" podID="423e27a8-c53a-47f3-9728-f9f41c9c4a9d" containerID="9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07" exitCode=0
Sep 16 01:15:52.052385 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.052281    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"
Sep 16 01:15:52.052385 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.052326    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg" event={"ID":"423e27a8-c53a-47f3-9728-f9f41c9c4a9d","Type":"ContainerDied","Data":"9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07"}
Sep 16 01:15:52.052508 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.052384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg" event={"ID":"423e27a8-c53a-47f3-9728-f9f41c9c4a9d","Type":"ContainerDied","Data":"cdec2a8f7f3d7182b345837a6c2105f5cbddc1298331cff016bb446e9f9a67cd"}
Sep 16 01:15:52.052508 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.052413    2573 scope.go:117] "RemoveContainer" containerID="9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07"
Sep 16 01:15:52.062574 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.062553    2573 scope.go:117] "RemoveContainer" containerID="9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07"
Sep 16 01:15:52.062839 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:15:52.062819    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07\": container with ID starting with 9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07 not found: ID does not exist" containerID="9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07"
Sep 16 01:15:52.062918 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.062847    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07"} err="failed to get container status \"9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07\": rpc error: code = NotFound desc = could not find container \"9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07\": container with ID starting with 9147cbf5bcd4822504df621cdef2502315ce1c4e51ecc023a21d12cfb956ed07 not found: ID does not exist"
Sep 16 01:15:52.073025 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.073002    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"]
Sep 16 01:15:52.076506 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.076486    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad13e5-64d48d9c7d-dvgxg"]
Sep 16 01:15:52.100307 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.100253    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:15:52.100307 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.100278    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chnmz\" (UniqueName: \"kubernetes.io/projected/423e27a8-c53a-47f3-9728-f9f41c9c4a9d-kube-api-access-chnmz\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:15:52.771716 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.771683    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423e27a8-c53a-47f3-9728-f9f41c9c4a9d" path="/var/lib/kubelet/pods/423e27a8-c53a-47f3-9728-f9f41c9c4a9d/volumes"
Sep 16 01:15:52.772085 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:52.772052    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5095b8fa-00d4-4e60-9931-ad9f51da67dc" path="/var/lib/kubelet/pods/5095b8fa-00d4-4e60-9931-ad9f51da67dc/volumes"
Sep 16 01:15:58.623974 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.623939    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"]
Sep 16 01:15:58.624422 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.624410    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423e27a8-c53a-47f3-9728-f9f41c9c4a9d" containerName="payload-pre-processing"
Sep 16 01:15:58.624466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.624424    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="423e27a8-c53a-47f3-9728-f9f41c9c4a9d" containerName="payload-pre-processing"
Sep 16 01:15:58.624466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.624439    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5095b8fa-00d4-4e60-9931-ad9f51da67dc" containerName="revoke-keys"
Sep 16 01:15:58.624466 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.624445    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5095b8fa-00d4-4e60-9931-ad9f51da67dc" containerName="revoke-keys"
Sep 16 01:15:58.624609 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.624538    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="423e27a8-c53a-47f3-9728-f9f41c9c4a9d" containerName="payload-pre-processing"
Sep 16 01:15:58.624609 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.624547    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5095b8fa-00d4-4e60-9931-ad9f51da67dc" containerName="revoke-keys"
Sep 16 01:15:58.627563 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.627511    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"
Sep 16 01:15:58.629683 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.629656    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:15:58.629804 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.629686    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:15:58.634780 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.634756    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"]
Sep 16 01:15:58.772179 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.772150    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88685d10-1237-4ec7-b5f2-a49cdacfb190-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q\" (UID: \"88685d10-1237-4ec7-b5f2-a49cdacfb190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"
Sep 16 01:15:58.874031 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.873940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88685d10-1237-4ec7-b5f2-a49cdacfb190-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q\" (UID: \"88685d10-1237-4ec7-b5f2-a49cdacfb190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"
Sep 16 01:15:58.874494 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.874476    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88685d10-1237-4ec7-b5f2-a49cdacfb190-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q\" (UID: \"88685d10-1237-4ec7-b5f2-a49cdacfb190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"
Sep 16 01:15:58.940386 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:58.940355    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"
Sep 16 01:15:59.211324 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:15:59.211298    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"]
Sep 16 01:15:59.212970 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:15:59.212945    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88685d10_1237_4ec7_b5f2_a49cdacfb190.slice/crio-d5559399cee30e5713205703a100bd1c4699b5560375b9e4cfcd8b1e9b31f9b2 WatchSource:0}: Error finding container d5559399cee30e5713205703a100bd1c4699b5560375b9e4cfcd8b1e9b31f9b2: Status 404 returned error can't find the container with id d5559399cee30e5713205703a100bd1c4699b5560375b9e4cfcd8b1e9b31f9b2
Sep 16 01:16:00.113606 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:00.113566    2573 generic.go:356] "Generic (PLEG): container finished" podID="88685d10-1237-4ec7-b5f2-a49cdacfb190" containerID="a6a484cd541a783859c2f0d546d472a84d1b8c007b3a89120c6ca06a63a63a1f" exitCode=0
Sep 16 01:16:00.114025 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:00.113625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q" event={"ID":"88685d10-1237-4ec7-b5f2-a49cdacfb190","Type":"ContainerDied","Data":"a6a484cd541a783859c2f0d546d472a84d1b8c007b3a89120c6ca06a63a63a1f"}
Sep 16 01:16:00.114025 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:00.113712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q" event={"ID":"88685d10-1237-4ec7-b5f2-a49cdacfb190","Type":"ContainerStarted","Data":"d5559399cee30e5713205703a100bd1c4699b5560375b9e4cfcd8b1e9b31f9b2"}
Sep 16 01:16:01.292031 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:01.292003    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"
Sep 16 01:16:01.399069 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:01.398963    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88685d10-1237-4ec7-b5f2-a49cdacfb190-maas-api-service-ca\") pod \"88685d10-1237-4ec7-b5f2-a49cdacfb190\" (UID: \"88685d10-1237-4ec7-b5f2-a49cdacfb190\") "
Sep 16 01:16:01.400198 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:01.399643    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88685d10-1237-4ec7-b5f2-a49cdacfb190-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88685d10-1237-4ec7-b5f2-a49cdacfb190" (UID: "88685d10-1237-4ec7-b5f2-a49cdacfb190"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:01.500823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:01.500786    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88685d10-1237-4ec7-b5f2-a49cdacfb190-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:02.127966 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:02.127933    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"
Sep 16 01:16:02.127966 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:02.127956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q" event={"ID":"88685d10-1237-4ec7-b5f2-a49cdacfb190","Type":"ContainerDied","Data":"d5559399cee30e5713205703a100bd1c4699b5560375b9e4cfcd8b1e9b31f9b2"}
Sep 16 01:16:02.128276 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:02.128012    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5559399cee30e5713205703a100bd1c4699b5560375b9e4cfcd8b1e9b31f9b2"
Sep 16 01:16:14.141605 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.141572    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"]
Sep 16 01:16:14.142117 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.141873    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c" podUID="da2207b6-7855-4c0b-83ca-3351ff0aa073" containerName="payload-pre-processing" containerID="cri-o://c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75" gracePeriod=30
Sep 16 01:16:14.293737 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.293693    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"]
Sep 16 01:16:14.297126 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.297098    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-00f5c9-tp54q"]
Sep 16 01:16:14.454369 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.454344    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:16:14.531440 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.530347    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2207b6-7855-4c0b-83ca-3351ff0aa073-plugins-config-volume\") pod \"da2207b6-7855-4c0b-83ca-3351ff0aa073\" (UID: \"da2207b6-7855-4c0b-83ca-3351ff0aa073\") "
Sep 16 01:16:14.531440 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.530418    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57pjb\" (UniqueName: \"kubernetes.io/projected/da2207b6-7855-4c0b-83ca-3351ff0aa073-kube-api-access-57pjb\") pod \"da2207b6-7855-4c0b-83ca-3351ff0aa073\" (UID: \"da2207b6-7855-4c0b-83ca-3351ff0aa073\") "
Sep 16 01:16:14.531440 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.531000    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2207b6-7855-4c0b-83ca-3351ff0aa073-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da2207b6-7855-4c0b-83ca-3351ff0aa073" (UID: "da2207b6-7855-4c0b-83ca-3351ff0aa073"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:14.533784 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.533735    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2207b6-7855-4c0b-83ca-3351ff0aa073-kube-api-access-57pjb" (OuterVolumeSpecName: "kube-api-access-57pjb") pod "da2207b6-7855-4c0b-83ca-3351ff0aa073" (UID: "da2207b6-7855-4c0b-83ca-3351ff0aa073"). InnerVolumeSpecName "kube-api-access-57pjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:14.632557 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.632492    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2207b6-7855-4c0b-83ca-3351ff0aa073-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:14.632557 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.632550    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pjb\" (UniqueName: \"kubernetes.io/projected/da2207b6-7855-4c0b-83ca-3351ff0aa073-kube-api-access-57pjb\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:14.771773 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:14.771739    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88685d10-1237-4ec7-b5f2-a49cdacfb190" path="/var/lib/kubelet/pods/88685d10-1237-4ec7-b5f2-a49cdacfb190/volumes"
Sep 16 01:16:15.211807 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.211775    2573 generic.go:356] "Generic (PLEG): container finished" podID="da2207b6-7855-4c0b-83ca-3351ff0aa073" containerID="c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75" exitCode=0
Sep 16 01:16:15.212234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.211817    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"
Sep 16 01:16:15.212234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.211867    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c" event={"ID":"da2207b6-7855-4c0b-83ca-3351ff0aa073","Type":"ContainerDied","Data":"c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75"}
Sep 16 01:16:15.212234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.211918    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c" event={"ID":"da2207b6-7855-4c0b-83ca-3351ff0aa073","Type":"ContainerDied","Data":"c068e99b54a34ea018d2d30d4ecb624d53afdf1ad0d756966265c47d1d8f43f3"}
Sep 16 01:16:15.212234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.211941    2573 scope.go:117] "RemoveContainer" containerID="c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75"
Sep 16 01:16:15.221637 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.221621    2573 scope.go:117] "RemoveContainer" containerID="c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75"
Sep 16 01:16:15.221891 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:16:15.221873    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75\": container with ID starting with c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75 not found: ID does not exist" containerID="c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75"
Sep 16 01:16:15.221933 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.221901    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75"} err="failed to get container status \"c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75\": rpc error: code = NotFound desc = could not find container \"c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75\": container with ID starting with c3da48353acc0620185c91f4ac7825d864940f464a3a53ad69963ce7e6a06a75 not found: ID does not exist"
Sep 16 01:16:15.248325 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.248246    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"]
Sep 16 01:16:15.251400 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:15.251379    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00f5c9-7f5cddb44b-rqx4c"]
Sep 16 01:16:16.772566 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:16.772501    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2207b6-7855-4c0b-83ca-3351ff0aa073" path="/var/lib/kubelet/pods/da2207b6-7855-4c0b-83ca-3351ff0aa073/volumes"
Sep 16 01:16:20.493814 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.493781    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"]
Sep 16 01:16:20.494251 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.494030    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy" containerID="cri-o://d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745" gracePeriod=30
Sep 16 01:16:20.848567 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.848477    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"]
Sep 16 01:16:20.848992 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.848978    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88685d10-1237-4ec7-b5f2-a49cdacfb190" containerName="revoke-keys"
Sep 16 01:16:20.849039 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.848994    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="88685d10-1237-4ec7-b5f2-a49cdacfb190" containerName="revoke-keys"
Sep 16 01:16:20.849039 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.849003    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2207b6-7855-4c0b-83ca-3351ff0aa073" containerName="payload-pre-processing"
Sep 16 01:16:20.849039 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.849008    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2207b6-7855-4c0b-83ca-3351ff0aa073" containerName="payload-pre-processing"
Sep 16 01:16:20.849139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.849075    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2207b6-7855-4c0b-83ca-3351ff0aa073" containerName="payload-pre-processing"
Sep 16 01:16:20.849139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.849086    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="88685d10-1237-4ec7-b5f2-a49cdacfb190" containerName="revoke-keys"
Sep 16 01:16:20.851808 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.851790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"
Sep 16 01:16:20.853825 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.853796    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:16:20.854020 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.854006    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:16:20.858486 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.858463    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"]
Sep 16 01:16:20.894099 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.894054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf218ba-3ce9-4600-a26e-3fd1682c9cc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn\" (UID: \"0bf218ba-3ce9-4600-a26e-3fd1682c9cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"
Sep 16 01:16:20.994866 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.994826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf218ba-3ce9-4600-a26e-3fd1682c9cc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn\" (UID: \"0bf218ba-3ce9-4600-a26e-3fd1682c9cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"
Sep 16 01:16:20.995430 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:20.995411    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf218ba-3ce9-4600-a26e-3fd1682c9cc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn\" (UID: \"0bf218ba-3ce9-4600-a26e-3fd1682c9cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"
Sep 16 01:16:21.163410 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:21.163322    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"
Sep 16 01:16:21.435474 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:21.435441    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"]
Sep 16 01:16:21.437056 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:16:21.437030    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf218ba_3ce9_4600_a26e_3fd1682c9cc4.slice/crio-63d74b4c9b029f4c0f8f5e827fa9334d4aab06bb26b016177a590f51a138ffae WatchSource:0}: Error finding container 63d74b4c9b029f4c0f8f5e827fa9334d4aab06bb26b016177a590f51a138ffae: Status 404 returned error can't find the container with id 63d74b4c9b029f4c0f8f5e827fa9334d4aab06bb26b016177a590f51a138ffae
Sep 16 01:16:22.261165 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:22.261124    2573 generic.go:356] "Generic (PLEG): container finished" podID="0bf218ba-3ce9-4600-a26e-3fd1682c9cc4" containerID="d9ba211788590a9b7bb530abd033f20e61a17f7314a6087b2c5d4829b65972fa" exitCode=0
Sep 16 01:16:22.261563 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:22.261209    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn" event={"ID":"0bf218ba-3ce9-4600-a26e-3fd1682c9cc4","Type":"ContainerDied","Data":"d9ba211788590a9b7bb530abd033f20e61a17f7314a6087b2c5d4829b65972fa"}
Sep 16 01:16:22.261563 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:22.261265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn" event={"ID":"0bf218ba-3ce9-4600-a26e-3fd1682c9cc4","Type":"ContainerStarted","Data":"63d74b4c9b029f4c0f8f5e827fa9334d4aab06bb26b016177a590f51a138ffae"}
Sep 16 01:16:23.418796 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:23.418774    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"
Sep 16 01:16:23.520979 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:23.520881    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf218ba-3ce9-4600-a26e-3fd1682c9cc4-maas-api-service-ca\") pod \"0bf218ba-3ce9-4600-a26e-3fd1682c9cc4\" (UID: \"0bf218ba-3ce9-4600-a26e-3fd1682c9cc4\") "
Sep 16 01:16:23.521246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:23.521222    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf218ba-3ce9-4600-a26e-3fd1682c9cc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bf218ba-3ce9-4600-a26e-3fd1682c9cc4" (UID: "0bf218ba-3ce9-4600-a26e-3fd1682c9cc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:23.622145 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:23.622112    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf218ba-3ce9-4600-a26e-3fd1682c9cc4-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:24.274674 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:24.274635    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"
Sep 16 01:16:24.274674 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:24.274651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn" event={"ID":"0bf218ba-3ce9-4600-a26e-3fd1682c9cc4","Type":"ContainerDied","Data":"63d74b4c9b029f4c0f8f5e827fa9334d4aab06bb26b016177a590f51a138ffae"}
Sep 16 01:16:24.274934 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:24.274691    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d74b4c9b029f4c0f8f5e827fa9334d4aab06bb26b016177a590f51a138ffae"
Sep 16 01:16:25.759647 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.759594    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:16:25.846323 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846294    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-socket\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846480 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846342    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krdrv\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-kube-api-access-krdrv\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846480 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846468    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-certs\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846612 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846509    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-credential-socket\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846612 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846585    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-envoy\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846612 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846608    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-token\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846614    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:25.846777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846639    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-podinfo\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846677    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-data\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846733    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e99831-2a48-46ae-aa89-a42ae583dee1-istiod-ca-cert\") pod \"17e99831-2a48-46ae-aa89-a42ae583dee1\" (UID: \"17e99831-2a48-46ae-aa89-a42ae583dee1\") "
Sep 16 01:16:25.846777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846749    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:25.847050 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846799    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:25.847050 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.846956    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-data" (OuterVolumeSpecName: "istio-data") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:25.847215 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.847195    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e99831-2a48-46ae-aa89-a42ae583dee1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:25.847292 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.847243    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:25.847292 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.847262    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:25.847292 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.847275    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:25.847292 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.847288    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:25.849107 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.849081    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:25.849239 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.849211    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:16:25.849300 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.849270    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-token" (OuterVolumeSpecName: "istio-token") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:25.849401 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.849386    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-kube-api-access-krdrv" (OuterVolumeSpecName: "kube-api-access-krdrv") pod "17e99831-2a48-46ae-aa89-a42ae583dee1" (UID: "17e99831-2a48-46ae-aa89-a42ae583dee1"). InnerVolumeSpecName "kube-api-access-krdrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:25.947795 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.947771    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:25.947795 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.947793    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:25.947935 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.947802    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e99831-2a48-46ae-aa89-a42ae583dee1-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:25.947935 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.947809    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e99831-2a48-46ae-aa89-a42ae583dee1-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:25.947935 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:25.947817    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krdrv\" (UniqueName: \"kubernetes.io/projected/17e99831-2a48-46ae-aa89-a42ae583dee1-kube-api-access-krdrv\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:26.289070 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.289039    2573 generic.go:356] "Generic (PLEG): container finished" podID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerID="d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745" exitCode=0
Sep 16 01:16:26.289237 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.289145    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"
Sep 16 01:16:26.289237 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.289172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" event={"ID":"17e99831-2a48-46ae-aa89-a42ae583dee1","Type":"ContainerDied","Data":"d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745"}
Sep 16 01:16:26.289338 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.289240    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz" event={"ID":"17e99831-2a48-46ae-aa89-a42ae583dee1","Type":"ContainerDied","Data":"6bd3fabd80d6feccb18dabb7f273fe9574645d5c1cb3bec36895b2f4263fdefc"}
Sep 16 01:16:26.289338 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.289276    2573 scope.go:117] "RemoveContainer" containerID="d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745"
Sep 16 01:16:26.298919 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.298904    2573 scope.go:117] "RemoveContainer" containerID="bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0"
Sep 16 01:16:26.307945 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.307929    2573 scope.go:117] "RemoveContainer" containerID="d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745"
Sep 16 01:16:26.308217 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:16:26.308167    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745\": container with ID starting with d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745 not found: ID does not exist" containerID="d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745"
Sep 16 01:16:26.308304 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.308256    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745"} err="failed to get container status \"d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745\": rpc error: code = NotFound desc = could not find container \"d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745\": container with ID starting with d56db1f3d417f62bb0a2209bd498213dd0c8da65f9b6dea7ae4846a55f9c3745 not found: ID does not exist"
Sep 16 01:16:26.308304 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.308281    2573 scope.go:117] "RemoveContainer" containerID="bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0"
Sep 16 01:16:26.309003 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:16:26.308975    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0\": container with ID starting with bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0 not found: ID does not exist" containerID="bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0"
Sep 16 01:16:26.309094 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.309009    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0"} err="failed to get container status \"bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0\": rpc error: code = NotFound desc = could not find container \"bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0\": container with ID starting with bcb401d232796ad605e33d2ecf9828b08cb2f25b7c79f8a55f164725aede5ab0 not found: ID does not exist"
Sep 16 01:16:26.311823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.311800    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"]
Sep 16 01:16:26.316810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.316792    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-00f5c9-openshift-default-5f4f8877d8-2j6wz"]
Sep 16 01:16:26.764702 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.764670    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:16:26.771286 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:26.771260    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" path="/var/lib/kubelet/pods/17e99831-2a48-46ae-aa89-a42ae583dee1/volumes"
Sep 16 01:16:31.733830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:31.733799    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"]
Sep 16 01:16:31.736389 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:31.736364    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f39e09-7qnxn"]
Sep 16 01:16:32.221626 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:32.221596    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2_27ceff0d-c76e-4065-ad4a-efb469675d40/payload-processing/0.log"
Sep 16 01:16:32.427969 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:32.427935    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-87r96_b3056300-83f7-42c9-97ac-82fff85a22b8/payload-processing/0.log"
Sep 16 01:16:32.547684 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:32.547607    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-87r96_b3056300-83f7-42c9-97ac-82fff85a22b8/payload-processing/0.log"
Sep 16 01:16:32.771991 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:32.771948    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf218ba-3ce9-4600-a26e-3fd1682c9cc4" path="/var/lib/kubelet/pods/0bf218ba-3ce9-4600-a26e-3fd1682c9cc4/volumes"
Sep 16 01:16:35.105676 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:35.105635    2573 scope.go:117] "RemoveContainer" containerID="87b52bca813a2881dc621c4ee0060dd3c78bec4d6971c12cdee080735cd1dbb7"
Sep 16 01:16:35.114595 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:35.114573    2573 scope.go:117] "RemoveContainer" containerID="8314db765cfd1f2223c23484a589eda836c44c941792a70ec57813fdc5214495"
Sep 16 01:16:38.266700 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.266654    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"]
Sep 16 01:16:38.267170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.266957    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy" containerID="cri-o://e175a257c52da91b4b7a618362b809ea7adcdfb8273db68ca6034c12d769fd89" gracePeriod=30
Sep 16 01:16:38.668837 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.668756    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"]
Sep 16 01:16:38.669237 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669224    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy"
Sep 16 01:16:38.669290 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669240    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy"
Sep 16 01:16:38.669290 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669253    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf218ba-3ce9-4600-a26e-3fd1682c9cc4" containerName="revoke-keys"
Sep 16 01:16:38.669290 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669260    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf218ba-3ce9-4600-a26e-3fd1682c9cc4" containerName="revoke-keys"
Sep 16 01:16:38.669290 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669273    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy"
Sep 16 01:16:38.669290 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669279    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy"
Sep 16 01:16:38.669468 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669339    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy"
Sep 16 01:16:38.669468 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669349    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e99831-2a48-46ae-aa89-a42ae583dee1" containerName="istio-proxy"
Sep 16 01:16:38.669468 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.669356    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf218ba-3ce9-4600-a26e-3fd1682c9cc4" containerName="revoke-keys"
Sep 16 01:16:38.671438 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.671421    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"
Sep 16 01:16:38.673475 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.673454    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:16:38.673640 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.673621    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:16:38.678885 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.678864    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"]
Sep 16 01:16:38.784753 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.784718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4371f2-5352-42d4-8e5e-7064b26e43c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg\" (UID: \"eb4371f2-5352-42d4-8e5e-7064b26e43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"
Sep 16 01:16:38.886306 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.886273    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4371f2-5352-42d4-8e5e-7064b26e43c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg\" (UID: \"eb4371f2-5352-42d4-8e5e-7064b26e43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"
Sep 16 01:16:38.886945 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.886925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4371f2-5352-42d4-8e5e-7064b26e43c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg\" (UID: \"eb4371f2-5352-42d4-8e5e-7064b26e43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"
Sep 16 01:16:38.983393 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:38.983369    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"
Sep 16 01:16:39.254908 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:39.254880    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"]
Sep 16 01:16:39.256250 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:16:39.256224    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4371f2_5352_42d4_8e5e_7064b26e43c5.slice/crio-1c321ac996046705b670322a43b72ebdbadfc20d13044bb2527d2f36dcd7a31b WatchSource:0}: Error finding container 1c321ac996046705b670322a43b72ebdbadfc20d13044bb2527d2f36dcd7a31b: Status 404 returned error can't find the container with id 1c321ac996046705b670322a43b72ebdbadfc20d13044bb2527d2f36dcd7a31b
Sep 16 01:16:39.389845 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:39.389813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg" event={"ID":"eb4371f2-5352-42d4-8e5e-7064b26e43c5","Type":"ContainerStarted","Data":"1c321ac996046705b670322a43b72ebdbadfc20d13044bb2527d2f36dcd7a31b"}
Sep 16 01:16:40.397858 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:40.397819    2573 generic.go:356] "Generic (PLEG): container finished" podID="eb4371f2-5352-42d4-8e5e-7064b26e43c5" containerID="7422a28b8619b3506f54ae5d68b0080ea27491bde2504a6c229d24c0b4095d14" exitCode=0
Sep 16 01:16:40.398234 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:40.397907    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg" event={"ID":"eb4371f2-5352-42d4-8e5e-7064b26e43c5","Type":"ContainerDied","Data":"7422a28b8619b3506f54ae5d68b0080ea27491bde2504a6c229d24c0b4095d14"}
Sep 16 01:16:41.558238 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:41.558215    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"
Sep 16 01:16:41.619254 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:41.619220    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4371f2-5352-42d4-8e5e-7064b26e43c5-maas-api-service-ca\") pod \"eb4371f2-5352-42d4-8e5e-7064b26e43c5\" (UID: \"eb4371f2-5352-42d4-8e5e-7064b26e43c5\") "
Sep 16 01:16:41.619584 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:41.619562    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4371f2-5352-42d4-8e5e-7064b26e43c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb4371f2-5352-42d4-8e5e-7064b26e43c5" (UID: "eb4371f2-5352-42d4-8e5e-7064b26e43c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:41.720083 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:41.720055    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4371f2-5352-42d4-8e5e-7064b26e43c5-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:41.840299 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:41.840267    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 01:16:41.840500 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:41.840321    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 01:16:42.411996 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:42.411969    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"
Sep 16 01:16:42.412212 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:42.411998    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg" event={"ID":"eb4371f2-5352-42d4-8e5e-7064b26e43c5","Type":"ContainerDied","Data":"1c321ac996046705b670322a43b72ebdbadfc20d13044bb2527d2f36dcd7a31b"}
Sep 16 01:16:42.412212 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:42.412046    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c321ac996046705b670322a43b72ebdbadfc20d13044bb2527d2f36dcd7a31b"
Sep 16 01:16:43.420512 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.420481    2573 generic.go:356] "Generic (PLEG): container finished" podID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerID="e175a257c52da91b4b7a618362b809ea7adcdfb8273db68ca6034c12d769fd89" exitCode=0
Sep 16 01:16:43.420909 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.420557    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" event={"ID":"c9cba585-f20d-48aa-8be8-ee6e78974664","Type":"ContainerDied","Data":"e175a257c52da91b4b7a618362b809ea7adcdfb8273db68ca6034c12d769fd89"}
Sep 16 01:16:43.420909 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.420621    2573 scope.go:117] "RemoveContainer" containerID="b8c476e4ad31ac8cf0a20617102328a29ab14dfde1d9445611fa60c5c944da2b"
Sep 16 01:16:43.518852 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.518832    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:16:43.538349 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538277    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-envoy\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.538349 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538345    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-socket\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.538578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538394    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cba585-f20d-48aa-8be8-ee6e78974664-istiod-ca-cert\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.538578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538441    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-podinfo\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.538578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538470    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-credential-socket\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.538578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538503    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-token\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.538811 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538613    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-data\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.538811 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538651    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgp2v\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-kube-api-access-jgp2v\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.538811 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.538697    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-certs\") pod \"c9cba585-f20d-48aa-8be8-ee6e78974664\" (UID: \"c9cba585-f20d-48aa-8be8-ee6e78974664\") "
Sep 16 01:16:43.539347 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.539250    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cba585-f20d-48aa-8be8-ee6e78974664-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:43.539347 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.539265    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:43.540123 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.539482    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:43.540123 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.539755    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:43.540123 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.540009    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:43.541344 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.541267    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:16:43.542225 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.542195    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-kube-api-access-jgp2v" (OuterVolumeSpecName: "kube-api-access-jgp2v") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "kube-api-access-jgp2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:43.542622 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.542588    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:43.542758 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.542735    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9cba585-f20d-48aa-8be8-ee6e78974664" (UID: "c9cba585-f20d-48aa-8be8-ee6e78974664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:16:43.639631 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639603    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9cba585-f20d-48aa-8be8-ee6e78974664-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:43.639631 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639631    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:43.639810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639640    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:43.639810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639649    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:43.639810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639658    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:43.639810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639666    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgp2v\" (UniqueName: \"kubernetes.io/projected/c9cba585-f20d-48aa-8be8-ee6e78974664-kube-api-access-jgp2v\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:43.639810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639675    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:43.639810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639683    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:43.639810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:43.639691    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9cba585-f20d-48aa-8be8-ee6e78974664-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:44.427600 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:44.427572    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"
Sep 16 01:16:44.427600 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:44.427584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm" event={"ID":"c9cba585-f20d-48aa-8be8-ee6e78974664","Type":"ContainerDied","Data":"d5d089236d366ef98fc890f47d18e548de77e30e10fedc3506a1bf0afeac47ef"}
Sep 16 01:16:44.428170 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:44.427646    2573 scope.go:117] "RemoveContainer" containerID="e175a257c52da91b4b7a618362b809ea7adcdfb8273db68ca6034c12d769fd89"
Sep 16 01:16:44.450279 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:44.450249    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"]
Sep 16 01:16:44.452714 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:44.452692    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f39e09-openshift-default-85cf756ddd-9ldvm"]
Sep 16 01:16:44.771272 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:44.771239    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" path="/var/lib/kubelet/pods/c9cba585-f20d-48aa-8be8-ee6e78974664/volumes"
Sep 16 01:16:48.816396 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:48.816365    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"]
Sep 16 01:16:48.818714 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:48.818692    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-45446b-29825355-kltl8"]
Sep 16 01:16:48.847284 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:48.847254    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"]
Sep 16 01:16:48.847536 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:48.847487    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s" podUID="39c8d413-1fa0-4364-8b62-751d4603d1a6" containerName="payload-pre-processing" containerID="cri-o://a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c" gracePeriod=30
Sep 16 01:16:48.997578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:48.997544    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"]
Sep 16 01:16:48.999906 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:48.999884    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-45446b-xcmqg"]
Sep 16 01:16:49.131306 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.131276    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:16:49.192935 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.192882    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r95rb\" (UniqueName: \"kubernetes.io/projected/39c8d413-1fa0-4364-8b62-751d4603d1a6-kube-api-access-r95rb\") pod \"39c8d413-1fa0-4364-8b62-751d4603d1a6\" (UID: \"39c8d413-1fa0-4364-8b62-751d4603d1a6\") "
Sep 16 01:16:49.193100 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.193006    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c8d413-1fa0-4364-8b62-751d4603d1a6-plugins-config-volume\") pod \"39c8d413-1fa0-4364-8b62-751d4603d1a6\" (UID: \"39c8d413-1fa0-4364-8b62-751d4603d1a6\") "
Sep 16 01:16:49.193406 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.193367    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c8d413-1fa0-4364-8b62-751d4603d1a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39c8d413-1fa0-4364-8b62-751d4603d1a6" (UID: "39c8d413-1fa0-4364-8b62-751d4603d1a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:49.195840 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.195808    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c8d413-1fa0-4364-8b62-751d4603d1a6-kube-api-access-r95rb" (OuterVolumeSpecName: "kube-api-access-r95rb") pod "39c8d413-1fa0-4364-8b62-751d4603d1a6" (UID: "39c8d413-1fa0-4364-8b62-751d4603d1a6"). InnerVolumeSpecName "kube-api-access-r95rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:16:49.294298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.294272    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r95rb\" (UniqueName: \"kubernetes.io/projected/39c8d413-1fa0-4364-8b62-751d4603d1a6-kube-api-access-r95rb\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:49.294298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.294299    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c8d413-1fa0-4364-8b62-751d4603d1a6-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:16:49.462253 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.462219    2573 generic.go:356] "Generic (PLEG): container finished" podID="39c8d413-1fa0-4364-8b62-751d4603d1a6" containerID="a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c" exitCode=0
Sep 16 01:16:49.462399 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.462338    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"
Sep 16 01:16:49.462399 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.462353    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s" event={"ID":"39c8d413-1fa0-4364-8b62-751d4603d1a6","Type":"ContainerDied","Data":"a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c"}
Sep 16 01:16:49.462480 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.462428    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s" event={"ID":"39c8d413-1fa0-4364-8b62-751d4603d1a6","Type":"ContainerDied","Data":"56ab02f37a67f965d6ff3c3af06eb29d80efb7f9c68befd98cfb88134dcc9147"}
Sep 16 01:16:49.462480 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.462455    2573 scope.go:117] "RemoveContainer" containerID="a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c"
Sep 16 01:16:49.471786 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.471766    2573 scope.go:117] "RemoveContainer" containerID="a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c"
Sep 16 01:16:49.472065 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:16:49.472046    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c\": container with ID starting with a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c not found: ID does not exist" containerID="a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c"
Sep 16 01:16:49.472142 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.472071    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c"} err="failed to get container status \"a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c\": rpc error: code = NotFound desc = could not find container \"a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c\": container with ID starting with a86d4941d3916719685211e4833aeb03a441d47c6a8dc389c741db683019a05c not found: ID does not exist"
Sep 16 01:16:49.483260 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.483239    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"]
Sep 16 01:16:49.486333 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:49.486314    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-45446b-789f6d5898-z449s"]
Sep 16 01:16:50.772704 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:50.772676    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c8d413-1fa0-4364-8b62-751d4603d1a6" path="/var/lib/kubelet/pods/39c8d413-1fa0-4364-8b62-751d4603d1a6/volumes"
Sep 16 01:16:50.773073 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:50.772976    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16555e0-f6c2-43b1-9be4-21ad80cfa30d" path="/var/lib/kubelet/pods/e16555e0-f6c2-43b1-9be4-21ad80cfa30d/volumes"
Sep 16 01:16:50.773241 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:50.773228    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb4371f2-5352-42d4-8e5e-7064b26e43c5" path="/var/lib/kubelet/pods/eb4371f2-5352-42d4-8e5e-7064b26e43c5/volumes"
Sep 16 01:16:56.798091 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798059    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"]
Sep 16 01:16:56.798499 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798486    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4371f2-5352-42d4-8e5e-7064b26e43c5" containerName="revoke-keys"
Sep 16 01:16:56.798499 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798500    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4371f2-5352-42d4-8e5e-7064b26e43c5" containerName="revoke-keys"
Sep 16 01:16:56.798619 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798535    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy"
Sep 16 01:16:56.798619 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798544    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy"
Sep 16 01:16:56.798619 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798593    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c8d413-1fa0-4364-8b62-751d4603d1a6" containerName="payload-pre-processing"
Sep 16 01:16:56.798619 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798600    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c8d413-1fa0-4364-8b62-751d4603d1a6" containerName="payload-pre-processing"
Sep 16 01:16:56.798742 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798689    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c8d413-1fa0-4364-8b62-751d4603d1a6" containerName="payload-pre-processing"
Sep 16 01:16:56.798742 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798698    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy"
Sep 16 01:16:56.798742 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798705    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4371f2-5352-42d4-8e5e-7064b26e43c5" containerName="revoke-keys"
Sep 16 01:16:56.798742 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.798713    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy"
Sep 16 01:16:56.803610 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.803593    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"
Sep 16 01:16:56.805728 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.805701    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:16:56.805728 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.805727    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:16:56.807973 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.807948    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"]
Sep 16 01:16:56.873818 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.873780    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783fc1c0-9645-49fd-925f-ef4a425dd3b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw\" (UID: \"783fc1c0-9645-49fd-925f-ef4a425dd3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"
Sep 16 01:16:56.974463 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.974430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783fc1c0-9645-49fd-925f-ef4a425dd3b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw\" (UID: \"783fc1c0-9645-49fd-925f-ef4a425dd3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"
Sep 16 01:16:56.975042 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:56.975022    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783fc1c0-9645-49fd-925f-ef4a425dd3b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw\" (UID: \"783fc1c0-9645-49fd-925f-ef4a425dd3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"
Sep 16 01:16:57.117100 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:57.117017    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"
Sep 16 01:16:57.393512 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:57.393485    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"]
Sep 16 01:16:57.394708 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:16:57.394679    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783fc1c0_9645_49fd_925f_ef4a425dd3b3.slice/crio-d2e3c31e46732d40f0fee811007b12ef9e3a3d8c1c54d37bff23022e7f1e951c WatchSource:0}: Error finding container d2e3c31e46732d40f0fee811007b12ef9e3a3d8c1c54d37bff23022e7f1e951c: Status 404 returned error can't find the container with id d2e3c31e46732d40f0fee811007b12ef9e3a3d8c1c54d37bff23022e7f1e951c
Sep 16 01:16:57.515704 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:57.515674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw" event={"ID":"783fc1c0-9645-49fd-925f-ef4a425dd3b3","Type":"ContainerStarted","Data":"d2e3c31e46732d40f0fee811007b12ef9e3a3d8c1c54d37bff23022e7f1e951c"}
Sep 16 01:16:58.522570 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:58.522478    2573 generic.go:356] "Generic (PLEG): container finished" podID="783fc1c0-9645-49fd-925f-ef4a425dd3b3" containerID="88ea7a679883cbe7323a9b9ec8f909f3298a9fe36006c1d12f6f133f273a6a07" exitCode=0
Sep 16 01:16:58.522917 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:58.522571    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw" event={"ID":"783fc1c0-9645-49fd-925f-ef4a425dd3b3","Type":"ContainerDied","Data":"88ea7a679883cbe7323a9b9ec8f909f3298a9fe36006c1d12f6f133f273a6a07"}
Sep 16 01:16:59.681697 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:59.681667    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"
Sep 16 01:16:59.810840 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:59.810773    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783fc1c0-9645-49fd-925f-ef4a425dd3b3-maas-api-service-ca\") pod \"783fc1c0-9645-49fd-925f-ef4a425dd3b3\" (UID: \"783fc1c0-9645-49fd-925f-ef4a425dd3b3\") "
Sep 16 01:16:59.811106 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:59.811084    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783fc1c0-9645-49fd-925f-ef4a425dd3b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "783fc1c0-9645-49fd-925f-ef4a425dd3b3" (UID: "783fc1c0-9645-49fd-925f-ef4a425dd3b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:16:59.912897 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:16:59.912865    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783fc1c0-9645-49fd-925f-ef4a425dd3b3-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:17:00.537639 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:00.537606    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"
Sep 16 01:17:00.537828 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:00.537644    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw" event={"ID":"783fc1c0-9645-49fd-925f-ef4a425dd3b3","Type":"ContainerDied","Data":"d2e3c31e46732d40f0fee811007b12ef9e3a3d8c1c54d37bff23022e7f1e951c"}
Sep 16 01:17:00.537828 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:00.537698    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2e3c31e46732d40f0fee811007b12ef9e3a3d8c1c54d37bff23022e7f1e951c"
Sep 16 01:17:07.551964 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:07.551924    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"]
Sep 16 01:17:07.554390 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:07.554367    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f46871-rpwbw"]
Sep 16 01:17:07.765681 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:07.765645    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:17:08.772086 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:08.772055    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783fc1c0-9645-49fd-925f-ef4a425dd3b3" path="/var/lib/kubelet/pods/783fc1c0-9645-49fd-925f-ef4a425dd3b3/volumes"
Sep 16 01:17:17.709500 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.709463    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"]
Sep 16 01:17:17.709936 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.709922    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783fc1c0-9645-49fd-925f-ef4a425dd3b3" containerName="revoke-keys"
Sep 16 01:17:17.709936 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.709937    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="783fc1c0-9645-49fd-925f-ef4a425dd3b3" containerName="revoke-keys"
Sep 16 01:17:17.710012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.709952    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy"
Sep 16 01:17:17.710012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.709957    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9cba585-f20d-48aa-8be8-ee6e78974664" containerName="istio-proxy"
Sep 16 01:17:17.710076 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.710035    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="783fc1c0-9645-49fd-925f-ef4a425dd3b3" containerName="revoke-keys"
Sep 16 01:17:17.713507 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.713485    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"
Sep 16 01:17:17.715510 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.715489    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:17:17.716266 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.716242    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:17:17.719473 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.719447    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"]
Sep 16 01:17:17.790916 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.790881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a429791c-b680-4058-a39d-6f5ac2f7b07b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r\" (UID: \"a429791c-b680-4058-a39d-6f5ac2f7b07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"
Sep 16 01:17:17.892237 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.892202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a429791c-b680-4058-a39d-6f5ac2f7b07b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r\" (UID: \"a429791c-b680-4058-a39d-6f5ac2f7b07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"
Sep 16 01:17:17.892839 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:17.892820    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a429791c-b680-4058-a39d-6f5ac2f7b07b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r\" (UID: \"a429791c-b680-4058-a39d-6f5ac2f7b07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"
Sep 16 01:17:18.027703 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:18.027620    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"
Sep 16 01:17:18.301511 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:18.301487    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"]
Sep 16 01:17:18.303497 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:17:18.303470    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda429791c_b680_4058_a39d_6f5ac2f7b07b.slice/crio-371cb1ee68368d492c6458cfbf1f7b00238331ad2372c1b8d80a03e110089d58 WatchSource:0}: Error finding container 371cb1ee68368d492c6458cfbf1f7b00238331ad2372c1b8d80a03e110089d58: Status 404 returned error can't find the container with id 371cb1ee68368d492c6458cfbf1f7b00238331ad2372c1b8d80a03e110089d58
Sep 16 01:17:18.646733 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:18.646694    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r" event={"ID":"a429791c-b680-4058-a39d-6f5ac2f7b07b","Type":"ContainerStarted","Data":"fb74d79a6c4b12e12bdd99cbdd1ba893b0c84cb827edd88f2d1663346f9b2023"}
Sep 16 01:17:18.646846 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:18.646748    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r" event={"ID":"a429791c-b680-4058-a39d-6f5ac2f7b07b","Type":"ContainerStarted","Data":"371cb1ee68368d492c6458cfbf1f7b00238331ad2372c1b8d80a03e110089d58"}
Sep 16 01:17:18.680467 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:18.680420    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r" podStartSLOduration=1.680404682 podStartE2EDuration="1.680404682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:18.678130407 +0000 UTC m=+2024.380952602" watchObservedRunningTime="2026-09-16 01:17:18.680404682 +0000 UTC m=+2024.383226914"
Sep 16 01:17:19.662011 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:19.661974    2573 generic.go:356] "Generic (PLEG): container finished" podID="a429791c-b680-4058-a39d-6f5ac2f7b07b" containerID="fb74d79a6c4b12e12bdd99cbdd1ba893b0c84cb827edd88f2d1663346f9b2023" exitCode=0
Sep 16 01:17:19.662401 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:19.662062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r" event={"ID":"a429791c-b680-4058-a39d-6f5ac2f7b07b","Type":"ContainerDied","Data":"fb74d79a6c4b12e12bdd99cbdd1ba893b0c84cb827edd88f2d1663346f9b2023"}
Sep 16 01:17:20.809185 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:20.809159    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"
Sep 16 01:17:20.822415 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:20.822394    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a429791c-b680-4058-a39d-6f5ac2f7b07b-maas-api-service-ca\") pod \"a429791c-b680-4058-a39d-6f5ac2f7b07b\" (UID: \"a429791c-b680-4058-a39d-6f5ac2f7b07b\") "
Sep 16 01:17:20.822780 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:20.822753    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a429791c-b680-4058-a39d-6f5ac2f7b07b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a429791c-b680-4058-a39d-6f5ac2f7b07b" (UID: "a429791c-b680-4058-a39d-6f5ac2f7b07b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:17:20.924015 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:20.923944    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a429791c-b680-4058-a39d-6f5ac2f7b07b-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:17:21.675553 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:21.675514    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"
Sep 16 01:17:21.675731 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:21.675499    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r" event={"ID":"a429791c-b680-4058-a39d-6f5ac2f7b07b","Type":"ContainerDied","Data":"371cb1ee68368d492c6458cfbf1f7b00238331ad2372c1b8d80a03e110089d58"}
Sep 16 01:17:21.675731 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:21.675653    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371cb1ee68368d492c6458cfbf1f7b00238331ad2372c1b8d80a03e110089d58"
Sep 16 01:17:32.765073 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:32.765037    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:17:33.213193 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.213160    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"]
Sep 16 01:17:33.215499 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.215471    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-089563-29825355-w8n5l"]
Sep 16 01:17:33.235639 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.235609    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"]
Sep 16 01:17:33.235884 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.235857    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2" podUID="27ceff0d-c76e-4065-ad4a-efb469675d40" containerName="payload-processing" containerID="cri-o://2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38" gracePeriod=30
Sep 16 01:17:33.244535 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.244495    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"]
Sep 16 01:17:33.244837 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.244808    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq" podUID="fd589a41-4148-491c-8441-c9ab7f546619" containerName="payload-pre-processing" containerID="cri-o://57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf" gracePeriod=30
Sep 16 01:17:33.411229 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.411194    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"]
Sep 16 01:17:33.413545 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.413474    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-089563-d8h2r"]
Sep 16 01:17:33.552949 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.552926    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:17:33.556037 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.556005    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:17:33.640300 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.640267    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd589a41-4148-491c-8441-c9ab7f546619-plugins-config-volume\") pod \"fd589a41-4148-491c-8441-c9ab7f546619\" (UID: \"fd589a41-4148-491c-8441-c9ab7f546619\") "
Sep 16 01:17:33.640488 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.640317    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ceff0d-c76e-4065-ad4a-efb469675d40-plugins-config-volume\") pod \"27ceff0d-c76e-4065-ad4a-efb469675d40\" (UID: \"27ceff0d-c76e-4065-ad4a-efb469675d40\") "
Sep 16 01:17:33.640488 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.640371    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h774c\" (UniqueName: \"kubernetes.io/projected/27ceff0d-c76e-4065-ad4a-efb469675d40-kube-api-access-h774c\") pod \"27ceff0d-c76e-4065-ad4a-efb469675d40\" (UID: \"27ceff0d-c76e-4065-ad4a-efb469675d40\") "
Sep 16 01:17:33.640488 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.640427    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2g78\" (UniqueName: \"kubernetes.io/projected/fd589a41-4148-491c-8441-c9ab7f546619-kube-api-access-g2g78\") pod \"fd589a41-4148-491c-8441-c9ab7f546619\" (UID: \"fd589a41-4148-491c-8441-c9ab7f546619\") "
Sep 16 01:17:33.641252 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.641202    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd589a41-4148-491c-8441-c9ab7f546619-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd589a41-4148-491c-8441-c9ab7f546619" (UID: "fd589a41-4148-491c-8441-c9ab7f546619"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:17:33.641653 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.641389    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ceff0d-c76e-4065-ad4a-efb469675d40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27ceff0d-c76e-4065-ad4a-efb469675d40" (UID: "27ceff0d-c76e-4065-ad4a-efb469675d40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:17:33.643627 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.643541    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ceff0d-c76e-4065-ad4a-efb469675d40-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:17:33.643723 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.643647    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd589a41-4148-491c-8441-c9ab7f546619-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:17:33.649096 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.647105    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd589a41-4148-491c-8441-c9ab7f546619-kube-api-access-g2g78" (OuterVolumeSpecName: "kube-api-access-g2g78") pod "fd589a41-4148-491c-8441-c9ab7f546619" (UID: "fd589a41-4148-491c-8441-c9ab7f546619"). InnerVolumeSpecName "kube-api-access-g2g78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:17:33.662427 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.662368    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ceff0d-c76e-4065-ad4a-efb469675d40-kube-api-access-h774c" (OuterVolumeSpecName: "kube-api-access-h774c") pod "27ceff0d-c76e-4065-ad4a-efb469675d40" (UID: "27ceff0d-c76e-4065-ad4a-efb469675d40"). InnerVolumeSpecName "kube-api-access-h774c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:17:33.745294 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.745251    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h774c\" (UniqueName: \"kubernetes.io/projected/27ceff0d-c76e-4065-ad4a-efb469675d40-kube-api-access-h774c\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:17:33.745294 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.745292    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2g78\" (UniqueName: \"kubernetes.io/projected/fd589a41-4148-491c-8441-c9ab7f546619-kube-api-access-g2g78\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:17:33.746829 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.746797    2573 generic.go:356] "Generic (PLEG): container finished" podID="fd589a41-4148-491c-8441-c9ab7f546619" containerID="57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf" exitCode=0
Sep 16 01:17:33.747005 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.746895    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"
Sep 16 01:17:33.747005 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.746919    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq" event={"ID":"fd589a41-4148-491c-8441-c9ab7f546619","Type":"ContainerDied","Data":"57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf"}
Sep 16 01:17:33.747126 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.747003    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq" event={"ID":"fd589a41-4148-491c-8441-c9ab7f546619","Type":"ContainerDied","Data":"ebf2b22862dd43bd53991c1c13b6fe708d80b91f7ea755329b255e0522708913"}
Sep 16 01:17:33.747126 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.747032    2573 scope.go:117] "RemoveContainer" containerID="57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf"
Sep 16 01:17:33.748451 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.748431    2573 generic.go:356] "Generic (PLEG): container finished" podID="27ceff0d-c76e-4065-ad4a-efb469675d40" containerID="2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38" exitCode=0
Sep 16 01:17:33.748586 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.748496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2" event={"ID":"27ceff0d-c76e-4065-ad4a-efb469675d40","Type":"ContainerDied","Data":"2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38"}
Sep 16 01:17:33.748586 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.748509    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"
Sep 16 01:17:33.748586 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.748569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2" event={"ID":"27ceff0d-c76e-4065-ad4a-efb469675d40","Type":"ContainerDied","Data":"7842db9ad1b1fcba1e079dfa967deeadbe7411c433e0009906e06c4b55d1ff7f"}
Sep 16 01:17:33.759010 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.758993    2573 scope.go:117] "RemoveContainer" containerID="57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf"
Sep 16 01:17:33.759332 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:17:33.759315    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf\": container with ID starting with 57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf not found: ID does not exist" containerID="57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf"
Sep 16 01:17:33.759399 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.759342    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf"} err="failed to get container status \"57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf\": rpc error: code = NotFound desc = could not find container \"57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf\": container with ID starting with 57b3d8d55f4bebad1d652f8797c060670bd238b5b2f6b41d0c4bc680bc1275cf not found: ID does not exist"
Sep 16 01:17:33.759399 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.759365    2573 scope.go:117] "RemoveContainer" containerID="2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38"
Sep 16 01:17:33.769564 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.769512    2573 scope.go:117] "RemoveContainer" containerID="2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38"
Sep 16 01:17:33.769904 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:17:33.769864    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38\": container with ID starting with 2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38 not found: ID does not exist" containerID="2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38"
Sep 16 01:17:33.769904 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.769890    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38"} err="failed to get container status \"2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38\": rpc error: code = NotFound desc = could not find container \"2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38\": container with ID starting with 2de8ba0040ba8b1b951e9cb6621a7fad8c2b5cbb4d0071aef682bea7a7699d38 not found: ID does not exist"
Sep 16 01:17:33.777155 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.777099    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"]
Sep 16 01:17:33.783052 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.783030    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-089563-756db6b94-2ngrq"]
Sep 16 01:17:33.793284 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.793264    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"]
Sep 16 01:17:33.797356 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:33.797323    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-089563-6b887d48f5-2c4x2"]
Sep 16 01:17:34.772025 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:34.771994    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27ceff0d-c76e-4065-ad4a-efb469675d40" path="/var/lib/kubelet/pods/27ceff0d-c76e-4065-ad4a-efb469675d40/volumes"
Sep 16 01:17:34.772388 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:34.772330    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a429791c-b680-4058-a39d-6f5ac2f7b07b" path="/var/lib/kubelet/pods/a429791c-b680-4058-a39d-6f5ac2f7b07b/volumes"
Sep 16 01:17:34.772695 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:34.772682    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd589a41-4148-491c-8441-c9ab7f546619" path="/var/lib/kubelet/pods/fd589a41-4148-491c-8441-c9ab7f546619/volumes"
Sep 16 01:17:34.772964 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:17:34.772951    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff4ee8b4-0c31-4f9e-9768-32c486811489" path="/var/lib/kubelet/pods/ff4ee8b4-0c31-4f9e-9768-32c486811489/volumes"
Sep 16 01:18:20.765061 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:20.765023    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:18:21.378980 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.378950    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"]
Sep 16 01:18:21.379201 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.379177    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy" containerID="cri-o://5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3" gracePeriod=30
Sep 16 01:18:21.718391 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718356    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"]
Sep 16 01:18:21.718817 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718805    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ceff0d-c76e-4065-ad4a-efb469675d40" containerName="payload-processing"
Sep 16 01:18:21.718870 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718820    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ceff0d-c76e-4065-ad4a-efb469675d40" containerName="payload-processing"
Sep 16 01:18:21.718870 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718831    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd589a41-4148-491c-8441-c9ab7f546619" containerName="payload-pre-processing"
Sep 16 01:18:21.718870 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718837    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd589a41-4148-491c-8441-c9ab7f546619" containerName="payload-pre-processing"
Sep 16 01:18:21.718870 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718845    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a429791c-b680-4058-a39d-6f5ac2f7b07b" containerName="revoke-keys"
Sep 16 01:18:21.718870 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718851    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a429791c-b680-4058-a39d-6f5ac2f7b07b" containerName="revoke-keys"
Sep 16 01:18:21.719036 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718946    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a429791c-b680-4058-a39d-6f5ac2f7b07b" containerName="revoke-keys"
Sep 16 01:18:21.719036 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718955    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd589a41-4148-491c-8441-c9ab7f546619" containerName="payload-pre-processing"
Sep 16 01:18:21.719036 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.718964    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ceff0d-c76e-4065-ad4a-efb469675d40" containerName="payload-processing"
Sep 16 01:18:21.728078 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.728053    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"
Sep 16 01:18:21.729553 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.729494    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"]
Sep 16 01:18:21.730426 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.730397    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:18:21.730635 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.730613    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:18:21.790185 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.790155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1ae63a-3e9b-4122-bf83-f073d3c7672b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42\" (UID: \"fc1ae63a-3e9b-4122-bf83-f073d3c7672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"
Sep 16 01:18:21.890884 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.890856    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1ae63a-3e9b-4122-bf83-f073d3c7672b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42\" (UID: \"fc1ae63a-3e9b-4122-bf83-f073d3c7672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"
Sep 16 01:18:21.891462 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:21.891439    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1ae63a-3e9b-4122-bf83-f073d3c7672b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42\" (UID: \"fc1ae63a-3e9b-4122-bf83-f073d3c7672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"
Sep 16 01:18:22.042105 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:22.042037    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"
Sep 16 01:18:22.318617 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:22.318594    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"]
Sep 16 01:18:22.320389 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:18:22.320367    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1ae63a_3e9b_4122_bf83_f073d3c7672b.slice/crio-199c816a03c67080c7ff375fcd6c211125bf281aebf4bcc5c89770e25f9e7b93 WatchSource:0}: Error finding container 199c816a03c67080c7ff375fcd6c211125bf281aebf4bcc5c89770e25f9e7b93: Status 404 returned error can't find the container with id 199c816a03c67080c7ff375fcd6c211125bf281aebf4bcc5c89770e25f9e7b93
Sep 16 01:18:23.037752 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:23.037722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42" event={"ID":"fc1ae63a-3e9b-4122-bf83-f073d3c7672b","Type":"ContainerStarted","Data":"ab56ecfdd99b6606f456af9b576eb2f3ed876635e7e80a0cd8b0b0940d6f2468"}
Sep 16 01:18:23.038207 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:23.037787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42" event={"ID":"fc1ae63a-3e9b-4122-bf83-f073d3c7672b","Type":"ContainerStarted","Data":"199c816a03c67080c7ff375fcd6c211125bf281aebf4bcc5c89770e25f9e7b93"}
Sep 16 01:18:23.050904 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:23.050857    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42" podStartSLOduration=2.050842143 podStartE2EDuration="2.050842143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:18:23.049840671 +0000 UTC m=+2088.752662866" watchObservedRunningTime="2026-09-16 01:18:23.050842143 +0000 UTC m=+2088.753664338"
Sep 16 01:18:24.044931 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:24.044889    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc1ae63a-3e9b-4122-bf83-f073d3c7672b" containerID="ab56ecfdd99b6606f456af9b576eb2f3ed876635e7e80a0cd8b0b0940d6f2468" exitCode=0
Sep 16 01:18:24.045344 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:24.044974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42" event={"ID":"fc1ae63a-3e9b-4122-bf83-f073d3c7672b","Type":"ContainerDied","Data":"ab56ecfdd99b6606f456af9b576eb2f3ed876635e7e80a0cd8b0b0940d6f2468"}
Sep 16 01:18:25.194843 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:25.194818    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"
Sep 16 01:18:25.326938 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:25.326850    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1ae63a-3e9b-4122-bf83-f073d3c7672b-maas-api-service-ca\") pod \"fc1ae63a-3e9b-4122-bf83-f073d3c7672b\" (UID: \"fc1ae63a-3e9b-4122-bf83-f073d3c7672b\") "
Sep 16 01:18:25.327164 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:25.327142    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1ae63a-3e9b-4122-bf83-f073d3c7672b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc1ae63a-3e9b-4122-bf83-f073d3c7672b" (UID: "fc1ae63a-3e9b-4122-bf83-f073d3c7672b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:18:25.327297 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:25.327282    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1ae63a-3e9b-4122-bf83-f073d3c7672b-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.057333 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.057304    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"
Sep 16 01:18:26.057597 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.057340    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42" event={"ID":"fc1ae63a-3e9b-4122-bf83-f073d3c7672b","Type":"ContainerDied","Data":"199c816a03c67080c7ff375fcd6c211125bf281aebf4bcc5c89770e25f9e7b93"}
Sep 16 01:18:26.057597 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.057391    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="199c816a03c67080c7ff375fcd6c211125bf281aebf4bcc5c89770e25f9e7b93"
Sep 16 01:18:26.625830 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.625809    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:18:26.740706 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.740679    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-data\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.740851 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.740726    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-socket\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.740851 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.740775    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-certs\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.740851 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.740794    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-token\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.740851 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.740843    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l26s\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-kube-api-access-8l26s\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.741048 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.740940    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-envoy\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.741048 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.740975    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-podinfo\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.741048 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741006    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istiod-ca-cert\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.741048 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.740970    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:26.741048 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741038    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-credential-socket\") pod \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\" (UID: \"ba3d51f7-e75a-4ed3-8496-0530c38e9776\") "
Sep 16 01:18:26.741310 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741081    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:26.741310 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741102    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:26.741429 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741353    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.741429 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741371    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.741429 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741378    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:18:26.741429 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741385    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.741832 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.741447    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:26.743353 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.743331    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-kube-api-access-8l26s" (OuterVolumeSpecName: "kube-api-access-8l26s") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "kube-api-access-8l26s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:18:26.743492 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.743402    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:18:26.743866 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.743845    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:26.743866 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.743851    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba3d51f7-e75a-4ed3-8496-0530c38e9776" (UID: "ba3d51f7-e75a-4ed3-8496-0530c38e9776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:18:26.842737 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.842709    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.842737 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.842734    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.842889 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.842748    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.842889 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.842759    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3d51f7-e75a-4ed3-8496-0530c38e9776-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.842889 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.842773    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:26.842889 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:26.842786    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l26s\" (UniqueName: \"kubernetes.io/projected/ba3d51f7-e75a-4ed3-8496-0530c38e9776-kube-api-access-8l26s\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:27.071583 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.071483    2573 generic.go:356] "Generic (PLEG): container finished" podID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerID="5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3" exitCode=0
Sep 16 01:18:27.071583 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.071571    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"
Sep 16 01:18:27.071759 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.071567    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" event={"ID":"ba3d51f7-e75a-4ed3-8496-0530c38e9776","Type":"ContainerDied","Data":"5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3"}
Sep 16 01:18:27.071759 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.071712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj" event={"ID":"ba3d51f7-e75a-4ed3-8496-0530c38e9776","Type":"ContainerDied","Data":"c33c77615d00e6e4b791996fee1e8a210b140040f1968457dc499ddb7510a543"}
Sep 16 01:18:27.071759 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.071734    2573 scope.go:117] "RemoveContainer" containerID="5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3"
Sep 16 01:18:27.081694 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.081676    2573 scope.go:117] "RemoveContainer" containerID="4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48"
Sep 16 01:18:27.088127 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.088089    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"]
Sep 16 01:18:27.092089 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.092072    2573 scope.go:117] "RemoveContainer" containerID="5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3"
Sep 16 01:18:27.092352 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.092333    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-089563-openshift-default-dd65bb8b4-gmmnj"]
Sep 16 01:18:27.092402 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:18:27.092363    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3\": container with ID starting with 5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3 not found: ID does not exist" containerID="5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3"
Sep 16 01:18:27.092402 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.092385    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3"} err="failed to get container status \"5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3\": rpc error: code = NotFound desc = could not find container \"5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3\": container with ID starting with 5f17c6e2ceb730c2e9f146b73423243c2cc6ed29cfc1624148f74251bb97ccf3 not found: ID does not exist"
Sep 16 01:18:27.092402 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.092401    2573 scope.go:117] "RemoveContainer" containerID="4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48"
Sep 16 01:18:27.092664 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:18:27.092647    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48\": container with ID starting with 4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48 not found: ID does not exist" containerID="4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48"
Sep 16 01:18:27.092715 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:27.092671    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48"} err="failed to get container status \"4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48\": rpc error: code = NotFound desc = could not find container \"4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48\": container with ID starting with 4be3965af55647b3aba7d26cc47e922de2b823a3f38abe3c67da07d2e2d5ab48 not found: ID does not exist"
Sep 16 01:18:28.770906 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:28.770868    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" path="/var/lib/kubelet/pods/ba3d51f7-e75a-4ed3-8496-0530c38e9776/volumes"
Sep 16 01:18:32.383410 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:32.383376    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"]
Sep 16 01:18:32.385899 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:32.385870    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-195f51-qzh42"]
Sep 16 01:18:32.771615 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:32.771582    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc1ae63a-3e9b-4122-bf83-f073d3c7672b" path="/var/lib/kubelet/pods/fc1ae63a-3e9b-4122-bf83-f073d3c7672b/volumes"
Sep 16 01:18:38.759738 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:38.759697    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"]
Sep 16 01:18:38.760260 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:38.759983    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy" containerID="cri-o://d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1" gracePeriod=30
Sep 16 01:18:39.141398 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.141314    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"]
Sep 16 01:18:39.141954 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.141939    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy"
Sep 16 01:18:39.142007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.141958    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy"
Sep 16 01:18:39.142007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.141980    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc1ae63a-3e9b-4122-bf83-f073d3c7672b" containerName="revoke-keys"
Sep 16 01:18:39.142007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.141988    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc1ae63a-3e9b-4122-bf83-f073d3c7672b" containerName="revoke-keys"
Sep 16 01:18:39.142007 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.142003    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy"
Sep 16 01:18:39.142139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.142012    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy"
Sep 16 01:18:39.142139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.142102    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc1ae63a-3e9b-4122-bf83-f073d3c7672b" containerName="revoke-keys"
Sep 16 01:18:39.142139 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.142113    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy"
Sep 16 01:18:39.145630 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.145609    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"
Sep 16 01:18:39.147915 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.147702    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:18:39.147915 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.147806    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:18:39.149861 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.149837    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"]
Sep 16 01:18:39.159124 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.159100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b41066-33db-4e0c-b1fb-e8856aae71c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf\" (UID: \"a3b41066-33db-4e0c-b1fb-e8856aae71c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"
Sep 16 01:18:39.260593 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.260564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b41066-33db-4e0c-b1fb-e8856aae71c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf\" (UID: \"a3b41066-33db-4e0c-b1fb-e8856aae71c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"
Sep 16 01:18:39.261178 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.261161    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b41066-33db-4e0c-b1fb-e8856aae71c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf\" (UID: \"a3b41066-33db-4e0c-b1fb-e8856aae71c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"
Sep 16 01:18:39.458832 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.458806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"
Sep 16 01:18:39.729422 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:39.729364    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"]
Sep 16 01:18:39.732662 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:18:39.732636    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b41066_33db_4e0c_b1fb_e8856aae71c1.slice/crio-a3590f7d27be7383650c5aec584a4b39934dab5b684076a640f806254b8537df WatchSource:0}: Error finding container a3590f7d27be7383650c5aec584a4b39934dab5b684076a640f806254b8537df: Status 404 returned error can't find the container with id a3590f7d27be7383650c5aec584a4b39934dab5b684076a640f806254b8537df
Sep 16 01:18:40.163245 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:40.163147    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf" event={"ID":"a3b41066-33db-4e0c-b1fb-e8856aae71c1","Type":"ContainerStarted","Data":"95fa5909b99eb255b08a98ebe6ec181455fb19b154293b838cf176e423f89c07"}
Sep 16 01:18:40.163245 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:40.163214    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf" event={"ID":"a3b41066-33db-4e0c-b1fb-e8856aae71c1","Type":"ContainerStarted","Data":"a3590f7d27be7383650c5aec584a4b39934dab5b684076a640f806254b8537df"}
Sep 16 01:18:40.181630 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:40.181579    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf" podStartSLOduration=1.181563349 podStartE2EDuration="1.181563349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:18:40.178725573 +0000 UTC m=+2105.881547779" watchObservedRunningTime="2026-09-16 01:18:40.181563349 +0000 UTC m=+2105.884385544"
Sep 16 01:18:41.170673 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:41.170636    2573 generic.go:356] "Generic (PLEG): container finished" podID="a3b41066-33db-4e0c-b1fb-e8856aae71c1" containerID="95fa5909b99eb255b08a98ebe6ec181455fb19b154293b838cf176e423f89c07" exitCode=0
Sep 16 01:18:41.171061 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:41.170722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf" event={"ID":"a3b41066-33db-4e0c-b1fb-e8856aae71c1","Type":"ContainerDied","Data":"95fa5909b99eb255b08a98ebe6ec181455fb19b154293b838cf176e423f89c07"}
Sep 16 01:18:42.314467 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:42.314440    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"
Sep 16 01:18:42.389279 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:42.389252    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b41066-33db-4e0c-b1fb-e8856aae71c1-maas-api-service-ca\") pod \"a3b41066-33db-4e0c-b1fb-e8856aae71c1\" (UID: \"a3b41066-33db-4e0c-b1fb-e8856aae71c1\") "
Sep 16 01:18:42.389597 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:42.389576    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b41066-33db-4e0c-b1fb-e8856aae71c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3b41066-33db-4e0c-b1fb-e8856aae71c1" (UID: "a3b41066-33db-4e0c-b1fb-e8856aae71c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:18:42.490330 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:42.490303    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b41066-33db-4e0c-b1fb-e8856aae71c1-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:43.183172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:43.183097    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"
Sep 16 01:18:43.183172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:43.183120    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf" event={"ID":"a3b41066-33db-4e0c-b1fb-e8856aae71c1","Type":"ContainerDied","Data":"a3590f7d27be7383650c5aec584a4b39934dab5b684076a640f806254b8537df"}
Sep 16 01:18:43.183172 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:43.183165    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3590f7d27be7383650c5aec584a4b39934dab5b684076a640f806254b8537df"
Sep 16 01:18:44.024491 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.024468    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:18:44.105050 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.104987    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-data\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105050 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105029    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istiod-ca-cert\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105253 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105063    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-token\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105253 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105092    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhxbw\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-kube-api-access-dhxbw\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105253 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105111    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-credential-socket\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105253 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105169    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-certs\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105253 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105210    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-socket\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105253 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105239    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-podinfo\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105596 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105283    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-envoy\") pod \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\" (UID: \"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7\") "
Sep 16 01:18:44.105660 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105273    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:44.105660 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105444    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:44.105660 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105465    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:44.105660 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105509    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:18:44.105660 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105515    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:44.105931 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105804    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.105985 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105950    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.105985 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105965    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.105985 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105979    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.106314 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.105993    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.107573 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.107496    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-kube-api-access-dhxbw" (OuterVolumeSpecName: "kube-api-access-dhxbw") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "kube-api-access-dhxbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:18:44.107573 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.107505    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:18:44.107734 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.107670    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:18:44.107777 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.107756    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" (UID: "9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:18:44.190275 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.190245    2573 generic.go:356] "Generic (PLEG): container finished" podID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerID="d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1" exitCode=0
Sep 16 01:18:44.190381 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.190302    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"
Sep 16 01:18:44.190381 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.190332    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" event={"ID":"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7","Type":"ContainerDied","Data":"d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1"}
Sep 16 01:18:44.190481 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.190385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t" event={"ID":"9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7","Type":"ContainerDied","Data":"f4fe1118dc3b465980f062574bc121469ad44332ebc81621fe072f7ec99c48e7"}
Sep 16 01:18:44.190481 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.190407    2573 scope.go:117] "RemoveContainer" containerID="d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1"
Sep 16 01:18:44.200694 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.200677    2573 scope.go:117] "RemoveContainer" containerID="fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34"
Sep 16 01:18:44.207531 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.207485    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.207690 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.207672    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhxbw\" (UniqueName: \"kubernetes.io/projected/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-kube-api-access-dhxbw\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.207818 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.207802    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.207943 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.207929    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:44.212342 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.212327    2573 scope.go:117] "RemoveContainer" containerID="d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1"
Sep 16 01:18:44.212654 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:18:44.212631    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1\": container with ID starting with d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1 not found: ID does not exist" containerID="d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1"
Sep 16 01:18:44.212739 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.212662    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1"} err="failed to get container status \"d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1\": rpc error: code = NotFound desc = could not find container \"d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1\": container with ID starting with d1df3c532e2ea5383c3273836f83200a96cb7ff3bf5fa73ac4c65b448b8357e1 not found: ID does not exist"
Sep 16 01:18:44.212739 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.212678    2573 scope.go:117] "RemoveContainer" containerID="fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34"
Sep 16 01:18:44.212866 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.212845    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"]
Sep 16 01:18:44.212942 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:18:44.212925    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34\": container with ID starting with fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34 not found: ID does not exist" containerID="fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34"
Sep 16 01:18:44.212983 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.212950    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34"} err="failed to get container status \"fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34\": rpc error: code = NotFound desc = could not find container \"fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34\": container with ID starting with fa7fb53ada1372d25724d0a06a74855a9cbafbe5c79f4f731a24acd3c0ccde34 not found: ID does not exist"
Sep 16 01:18:44.216282 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.216262    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-195f51-openshift-default-6cc5dfc99d-hfd9t"]
Sep 16 01:18:44.768109 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.768077    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:18:44.771482 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:44.771459    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" path="/var/lib/kubelet/pods/9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7/volumes"
Sep 16 01:18:49.610039 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.610004    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"]
Sep 16 01:18:49.610636 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.610270    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4" podUID="6fe07b1e-fe22-40a8-a367-abef6b6adaf0" containerName="payload-processing" containerID="cri-o://268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a" gracePeriod=30
Sep 16 01:18:49.796067 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.796025    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"]
Sep 16 01:18:49.799540 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.799494    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f831c5-lkpqf"]
Sep 16 01:18:49.879252 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.879230    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:18:49.959303 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.959269    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-plugins-config-volume\") pod \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\" (UID: \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\") "
Sep 16 01:18:49.959497 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.959443    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmr4p\" (UniqueName: \"kubernetes.io/projected/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-kube-api-access-pmr4p\") pod \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\" (UID: \"6fe07b1e-fe22-40a8-a367-abef6b6adaf0\") "
Sep 16 01:18:49.959774 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.959731    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fe07b1e-fe22-40a8-a367-abef6b6adaf0" (UID: "6fe07b1e-fe22-40a8-a367-abef6b6adaf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:18:49.961682 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:49.961656    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-kube-api-access-pmr4p" (OuterVolumeSpecName: "kube-api-access-pmr4p") pod "6fe07b1e-fe22-40a8-a367-abef6b6adaf0" (UID: "6fe07b1e-fe22-40a8-a367-abef6b6adaf0"). InnerVolumeSpecName "kube-api-access-pmr4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:18:50.060168 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.060142    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmr4p\" (UniqueName: \"kubernetes.io/projected/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-kube-api-access-pmr4p\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:50.060168 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.060164    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe07b1e-fe22-40a8-a367-abef6b6adaf0-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:18:50.225820 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.225784    2573 generic.go:356] "Generic (PLEG): container finished" podID="6fe07b1e-fe22-40a8-a367-abef6b6adaf0" containerID="268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a" exitCode=0
Sep 16 01:18:50.226012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.225835    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"
Sep 16 01:18:50.226012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.225870    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4" event={"ID":"6fe07b1e-fe22-40a8-a367-abef6b6adaf0","Type":"ContainerDied","Data":"268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a"}
Sep 16 01:18:50.226012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.225934    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4" event={"ID":"6fe07b1e-fe22-40a8-a367-abef6b6adaf0","Type":"ContainerDied","Data":"116f3fb70a32adefcbc53d9f315f49537fa1344af4e10ae9a43807cbad8642ab"}
Sep 16 01:18:50.226012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.225967    2573 scope.go:117] "RemoveContainer" containerID="268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a"
Sep 16 01:18:50.236472 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.236455    2573 scope.go:117] "RemoveContainer" containerID="268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a"
Sep 16 01:18:50.236760 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:18:50.236743    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a\": container with ID starting with 268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a not found: ID does not exist" containerID="268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a"
Sep 16 01:18:50.236818 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.236768    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a"} err="failed to get container status \"268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a\": rpc error: code = NotFound desc = could not find container \"268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a\": container with ID starting with 268fd415a4e5b7c01a5093039eea6f2463173100892c58c1fa46003dc390ed7a not found: ID does not exist"
Sep 16 01:18:50.247463 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.247439    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"]
Sep 16 01:18:50.250691 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.250670    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f831c5-bd474457c-p7hh4"]
Sep 16 01:18:50.770892 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.770861    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe07b1e-fe22-40a8-a367-abef6b6adaf0" path="/var/lib/kubelet/pods/6fe07b1e-fe22-40a8-a367-abef6b6adaf0/volumes"
Sep 16 01:18:50.771270 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:50.771199    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3b41066-33db-4e0c-b1fb-e8856aae71c1" path="/var/lib/kubelet/pods/a3b41066-33db-4e0c-b1fb-e8856aae71c1/volumes"
Sep 16 01:18:56.399354 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.399317    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"]
Sep 16 01:18:56.399738 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.399588    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy" containerID="cri-o://4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9" gracePeriod=30
Sep 16 01:18:56.791620 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.791589    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"]
Sep 16 01:18:56.791978 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.791965    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b41066-33db-4e0c-b1fb-e8856aae71c1" containerName="revoke-keys"
Sep 16 01:18:56.792019 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.791980    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b41066-33db-4e0c-b1fb-e8856aae71c1" containerName="revoke-keys"
Sep 16 01:18:56.792019 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.791987    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe07b1e-fe22-40a8-a367-abef6b6adaf0" containerName="payload-processing"
Sep 16 01:18:56.792019 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.791993    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe07b1e-fe22-40a8-a367-abef6b6adaf0" containerName="payload-processing"
Sep 16 01:18:56.792019 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792003    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy"
Sep 16 01:18:56.792019 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792010    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy"
Sep 16 01:18:56.792175 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792033    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy"
Sep 16 01:18:56.792175 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792038    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy"
Sep 16 01:18:56.792175 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792092    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3d51f7-e75a-4ed3-8496-0530c38e9776" containerName="istio-proxy"
Sep 16 01:18:56.792175 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792100    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b41066-33db-4e0c-b1fb-e8856aae71c1" containerName="revoke-keys"
Sep 16 01:18:56.792175 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792106    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe07b1e-fe22-40a8-a367-abef6b6adaf0" containerName="payload-processing"
Sep 16 01:18:56.792175 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792113    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy"
Sep 16 01:18:56.792175 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.792118    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce110ad-d891-4dfa-a4ac-b0fe85c9cdd7" containerName="istio-proxy"
Sep 16 01:18:56.796670 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.796653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"
Sep 16 01:18:56.799244 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.799092    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6blf\""
Sep 16 01:18:56.799244 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.799126    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 01:18:56.801845 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.801819    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"]
Sep 16 01:18:56.924567 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:56.924512    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23708c05-b8e7-41e1-aa1e-d8046892d4b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5\" (UID: \"23708c05-b8e7-41e1-aa1e-d8046892d4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"
Sep 16 01:18:57.025334 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:57.025299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23708c05-b8e7-41e1-aa1e-d8046892d4b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5\" (UID: \"23708c05-b8e7-41e1-aa1e-d8046892d4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"
Sep 16 01:18:57.025927 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:57.025908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23708c05-b8e7-41e1-aa1e-d8046892d4b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5\" (UID: \"23708c05-b8e7-41e1-aa1e-d8046892d4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"
Sep 16 01:18:57.109883 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:57.109794    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"
Sep 16 01:18:57.383138 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:57.383110    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"]
Sep 16 01:18:57.384810 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:18:57.384785    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23708c05_b8e7_41e1_aa1e_d8046892d4b0.slice/crio-8c6d23def4e7c0c3a6717b10867b557989be36a52d1b8ba18cb4f137960c15c3 WatchSource:0}: Error finding container 8c6d23def4e7c0c3a6717b10867b557989be36a52d1b8ba18cb4f137960c15c3: Status 404 returned error can't find the container with id 8c6d23def4e7c0c3a6717b10867b557989be36a52d1b8ba18cb4f137960c15c3
Sep 16 01:18:58.281824 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:58.281783    2573 generic.go:356] "Generic (PLEG): container finished" podID="23708c05-b8e7-41e1-aa1e-d8046892d4b0" containerID="846f8ab13ef575c0dcf8e7756d88391d8ff0dc362173b69158a5478dc875aab7" exitCode=0
Sep 16 01:18:58.282199 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:58.281865    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5" event={"ID":"23708c05-b8e7-41e1-aa1e-d8046892d4b0","Type":"ContainerDied","Data":"846f8ab13ef575c0dcf8e7756d88391d8ff0dc362173b69158a5478dc875aab7"}
Sep 16 01:18:58.282199 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:58.281942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5" event={"ID":"23708c05-b8e7-41e1-aa1e-d8046892d4b0","Type":"ContainerStarted","Data":"8c6d23def4e7c0c3a6717b10867b557989be36a52d1b8ba18cb4f137960c15c3"}
Sep 16 01:18:59.418403 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:59.418377    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"
Sep 16 01:18:59.551319 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:59.551241    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23708c05-b8e7-41e1-aa1e-d8046892d4b0-maas-api-service-ca\") pod \"23708c05-b8e7-41e1-aa1e-d8046892d4b0\" (UID: \"23708c05-b8e7-41e1-aa1e-d8046892d4b0\") "
Sep 16 01:18:59.551588 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:59.551565    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23708c05-b8e7-41e1-aa1e-d8046892d4b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23708c05-b8e7-41e1-aa1e-d8046892d4b0" (UID: "23708c05-b8e7-41e1-aa1e-d8046892d4b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:18:59.651957 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:18:59.651929    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23708c05-b8e7-41e1-aa1e-d8046892d4b0-maas-api-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:00.294686 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:00.294651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5" event={"ID":"23708c05-b8e7-41e1-aa1e-d8046892d4b0","Type":"ContainerDied","Data":"8c6d23def4e7c0c3a6717b10867b557989be36a52d1b8ba18cb4f137960c15c3"}
Sep 16 01:19:00.294855 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:00.294690    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"
Sep 16 01:19:00.294855 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:00.294698    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c6d23def4e7c0c3a6717b10867b557989be36a52d1b8ba18cb4f137960c15c3"
Sep 16 01:19:01.677350 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.677270    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:19:01.774065 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774034    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-token\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774087    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-certs\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774121    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-socket\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774165    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-data\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774211    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-envoy\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774246 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774233    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8459a3f-245f-42bd-ba92-29fc424c65fa-istiod-ca-cert\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774554 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774257    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-podinfo\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774554 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774301    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-credential-socket\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774554 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774353    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z79d7\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-kube-api-access-z79d7\") pod \"f8459a3f-245f-42bd-ba92-29fc424c65fa\" (UID: \"f8459a3f-245f-42bd-ba92-29fc424c65fa\") "
Sep 16 01:19:01.774554 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774433    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:01.774554 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774471    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:01.774554 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774490    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:01.774874 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774628    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8459a3f-245f-42bd-ba92-29fc424c65fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:19:01.774874 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774696    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:01.774874 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774696    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:01.774874 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774797    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:01.774874 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.774814    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:01.776683 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.776662    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:19:01.776794 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.776771    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-kube-api-access-z79d7" (OuterVolumeSpecName: "kube-api-access-z79d7") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "kube-api-access-z79d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:19:01.776956 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.776942    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:19:01.777062 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.777045    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8459a3f-245f-42bd-ba92-29fc424c65fa" (UID: "f8459a3f-245f-42bd-ba92-29fc424c65fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:01.875946 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.875913    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:01.876128 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.876036    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:01.876177 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.876163    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8459a3f-245f-42bd-ba92-29fc424c65fa-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:01.876223 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.876183    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8459a3f-245f-42bd-ba92-29fc424c65fa-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:01.876223 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.876192    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8459a3f-245f-42bd-ba92-29fc424c65fa-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:01.876223 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:01.876203    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79d7\" (UniqueName: \"kubernetes.io/projected/f8459a3f-245f-42bd-ba92-29fc424c65fa-kube-api-access-z79d7\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:02.308126 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.308093    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerID="4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9" exitCode=0
Sep 16 01:19:02.308304 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.308142    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"
Sep 16 01:19:02.308304 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.308178    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" event={"ID":"f8459a3f-245f-42bd-ba92-29fc424c65fa","Type":"ContainerDied","Data":"4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9"}
Sep 16 01:19:02.308304 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.308230    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r" event={"ID":"f8459a3f-245f-42bd-ba92-29fc424c65fa","Type":"ContainerDied","Data":"e1936bc956835861e4e03053686ee1bb9e94534bb62693eff1094422e0cff2a0"}
Sep 16 01:19:02.308304 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.308257    2573 scope.go:117] "RemoveContainer" containerID="4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9"
Sep 16 01:19:02.318304 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.318286    2573 scope.go:117] "RemoveContainer" containerID="beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3"
Sep 16 01:19:02.329019 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.328998    2573 scope.go:117] "RemoveContainer" containerID="4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9"
Sep 16 01:19:02.329441 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:19:02.329421    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9\": container with ID starting with 4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9 not found: ID does not exist" containerID="4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9"
Sep 16 01:19:02.329499 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.329453    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9"} err="failed to get container status \"4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9\": rpc error: code = NotFound desc = could not find container \"4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9\": container with ID starting with 4ecab39b93a29ae2915cc19ba64ea8908a4113cd4a904c9d9388e58590ee0ac9 not found: ID does not exist"
Sep 16 01:19:02.329499 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.329474    2573 scope.go:117] "RemoveContainer" containerID="beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3"
Sep 16 01:19:02.329873 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:19:02.329846    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3\": container with ID starting with beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3 not found: ID does not exist" containerID="beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3"
Sep 16 01:19:02.330011 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.329880    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3"} err="failed to get container status \"beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3\": rpc error: code = NotFound desc = could not find container \"beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3\": container with ID starting with beb28b4fa600c1eef90f3213cccea0cb6adb219695946658d924ea10f125b3c3 not found: ID does not exist"
Sep 16 01:19:02.330669 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.330648    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"]
Sep 16 01:19:02.332694 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.332672    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f831c5-openshift-default-648cfddf5d-8j74r"]
Sep 16 01:19:02.773149 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:02.773115    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" path="/var/lib/kubelet/pods/f8459a3f-245f-42bd-ba92-29fc424c65fa/volumes"
Sep 16 01:19:07.343589 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:07.343557    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"]
Sep 16 01:19:07.345243 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:07.345219    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ee1bf-29825355-qphkt"]
Sep 16 01:19:07.524865 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:07.524834    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"]
Sep 16 01:19:07.528019 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:07.527993    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ee1bf-ctjz5"]
Sep 16 01:19:08.770999 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:08.770963    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23708c05-b8e7-41e1-aa1e-d8046892d4b0" path="/var/lib/kubelet/pods/23708c05-b8e7-41e1-aa1e-d8046892d4b0/volumes"
Sep 16 01:19:08.771365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:08.771253    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cabd2813-0a40-410b-8ac5-8e794bd4ee9c" path="/var/lib/kubelet/pods/cabd2813-0a40-410b-8ac5-8e794bd4ee9c/volumes"
Sep 16 01:19:14.084811 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:14.084777    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"]
Sep 16 01:19:14.085269 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:14.085029    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy" containerID="cri-o://28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d" gracePeriod=30
Sep 16 01:19:19.369298 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.369220    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:19:19.420463 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.420431    2573 generic.go:356] "Generic (PLEG): container finished" podID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerID="28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d" exitCode=0
Sep 16 01:19:19.420653 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.420494    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"
Sep 16 01:19:19.420653 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.420555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" event={"ID":"f554aabb-aac0-4371-9934-1cdf2ea7d6ba","Type":"ContainerDied","Data":"28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d"}
Sep 16 01:19:19.420653 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.420622    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw" event={"ID":"f554aabb-aac0-4371-9934-1cdf2ea7d6ba","Type":"ContainerDied","Data":"3d6a6994537d61975d9a6acefe0efecd556e237221fd70dfbcf010437ab5d65c"}
Sep 16 01:19:19.420844 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.420654    2573 scope.go:117] "RemoveContainer" containerID="28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d"
Sep 16 01:19:19.430088 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430060    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-token\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430208 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430136    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-credential-socket\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430208 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430166    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-socket\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430330 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430227    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-envoy\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430330 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430268    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-podinfo\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430330 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430297    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istiod-ca-cert\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430570 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430329    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-data\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430570 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430380    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tpbl\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-kube-api-access-8tpbl\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430570 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430414    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-certs\") pod \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\" (UID: \"f554aabb-aac0-4371-9934-1cdf2ea7d6ba\") "
Sep 16 01:19:19.430570 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430458    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:19.430779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430717    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:19.430831 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430777    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-credential-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.430831 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.430794    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-data" (OuterVolumeSpecName: "istio-data") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:19.431059 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.431027    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:19.431262 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.431221    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:19:19.431330 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.431285    2573 scope.go:117] "RemoveContainer" containerID="336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433"
Sep 16 01:19:19.433356 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.433325    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:19:19.433356 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.433337    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:19:19.434066 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.434039    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-kube-api-access-8tpbl" (OuterVolumeSpecName: "kube-api-access-8tpbl") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "kube-api-access-8tpbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:19:19.434589 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.434557    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f554aabb-aac0-4371-9934-1cdf2ea7d6ba" (UID: "f554aabb-aac0-4371-9934-1cdf2ea7d6ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:19:19.452121 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.452103    2573 scope.go:117] "RemoveContainer" containerID="28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d"
Sep 16 01:19:19.452445 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:19:19.452428    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d\": container with ID starting with 28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d not found: ID does not exist" containerID="28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d"
Sep 16 01:19:19.452502 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.452453    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d"} err="failed to get container status \"28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d\": rpc error: code = NotFound desc = could not find container \"28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d\": container with ID starting with 28a79b43bd00a5450314f22265d4e6d2bda1e87e8e557192d34cc6e31283465d not found: ID does not exist"
Sep 16 01:19:19.452502 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.452471    2573 scope.go:117] "RemoveContainer" containerID="336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433"
Sep 16 01:19:19.452744 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:19:19.452732    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433\": container with ID starting with 336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433 not found: ID does not exist" containerID="336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433"
Sep 16 01:19:19.452784 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.452748    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433"} err="failed to get container status \"336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433\": rpc error: code = NotFound desc = could not find container \"336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433\": container with ID starting with 336d82654cbe30b6849b3607e64be3a0b1a43c8e505cf3f1e5f0db855db14433 not found: ID does not exist"
Sep 16 01:19:19.531929 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.531910    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-envoy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.531929 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.531931    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-podinfo\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.532044 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.531940    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istiod-ca-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.532044 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.531947    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-data\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.532044 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.531956    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tpbl\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-kube-api-access-8tpbl\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.532044 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.531964    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.532044 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.531972    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-istio-token\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.532044 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.531980    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f554aabb-aac0-4371-9934-1cdf2ea7d6ba-workload-socket\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:19:19.741586 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.741551    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"]
Sep 16 01:19:19.746030 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:19.746008    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ee1bf-openshift-default-9d6c8d686-5g8kw"]
Sep 16 01:19:20.771365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:20.771321    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" path="/var/lib/kubelet/pods/f554aabb-aac0-4371-9934-1cdf2ea7d6ba/volumes"
Sep 16 01:19:31.764638 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:31.764550    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:19:35.200502 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:35.200470    2573 scope.go:117] "RemoveContainer" containerID="f7b5386b0af581237b9253724ffe25eb921fde7616739c5befce92995db0d39a"
Sep 16 01:19:47.765097 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:19:47.765057    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:20:49.765066 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:20:49.765032    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:21:00.765057 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:21:00.764984    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:21:35.246408 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:21:35.246376    2573 scope.go:117] "RemoveContainer" containerID="2b5faf5c764aa91c24a69841cf7745e2fb6b14d6fa8abdcba4b6699f9fd9ec0d"
Sep 16 01:21:35.254896 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:21:35.254874    2573 scope.go:117] "RemoveContainer" containerID="d5fc4208e812ae5b2417301134a466e0fb165df1348916091e0ee42f87589d1d"
Sep 16 01:21:35.262798 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:21:35.262781    2573 scope.go:117] "RemoveContainer" containerID="eb2143100ba2e89308cdf55fe9932b85cc38c41e60739af0077cd6fd410c23b7"
Sep 16 01:21:55.765133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:21:55.765101    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:22:01.861721 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:01.861685    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"]
Sep 16 01:22:01.862267 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:01.861896    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw" podUID="0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c" containerName="manager" containerID="cri-o://3ee23b580a8a3f50d915c49b24886db543c64904a3ccb65415d3c10ec04a0a77" gracePeriod=10
Sep 16 01:22:02.343693 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.343653    2573 generic.go:356] "Generic (PLEG): container finished" podID="0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c" containerID="3ee23b580a8a3f50d915c49b24886db543c64904a3ccb65415d3c10ec04a0a77" exitCode=0
Sep 16 01:22:02.343862 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.343724    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw" event={"ID":"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c","Type":"ContainerDied","Data":"3ee23b580a8a3f50d915c49b24886db543c64904a3ccb65415d3c10ec04a0a77"}
Sep 16 01:22:02.715122 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.715092    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 01:22:02.859229 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.859141    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd55s\" (UniqueName: \"kubernetes.io/projected/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-kube-api-access-nd55s\") pod \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\" (UID: \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\") "
Sep 16 01:22:02.859229 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.859179    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-extensions-socket-volume\") pod \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\" (UID: \"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c\") "
Sep 16 01:22:02.859598 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.859574    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c" (UID: "0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:22:02.861408 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.861386    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-kube-api-access-nd55s" (OuterVolumeSpecName: "kube-api-access-nd55s") pod "0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c" (UID: "0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c"). InnerVolumeSpecName "kube-api-access-nd55s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:22:02.960238 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.960203    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd55s\" (UniqueName: \"kubernetes.io/projected/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-kube-api-access-nd55s\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:22:02.960238 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:02.960235    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c-extensions-socket-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:22:03.350451 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:03.350414    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"
Sep 16 01:22:03.350451 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:03.350441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw" event={"ID":"0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c","Type":"ContainerDied","Data":"ab6d86c49511ef9cd60bb3ce98c88b12613249d6845cc5af327e53e92df45434"}
Sep 16 01:22:03.350741 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:03.350495    2573 scope.go:117] "RemoveContainer" containerID="3ee23b580a8a3f50d915c49b24886db543c64904a3ccb65415d3c10ec04a0a77"
Sep 16 01:22:03.371219 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:03.371194    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"]
Sep 16 01:22:03.374603 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:03.374581    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mlnjw"]
Sep 16 01:22:04.771740 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:04.771708    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c" path="/var/lib/kubelet/pods/0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c/volumes"
Sep 16 01:22:08.765532 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:08.765489    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:22:35.292648 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:35.292576    2573 scope.go:117] "RemoveContainer" containerID="d9ba211788590a9b7bb530abd033f20e61a17f7314a6087b2c5d4829b65972fa"
Sep 16 01:22:35.301271 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:35.301253    2573 scope.go:117] "RemoveContainer" containerID="7e8d62afe8cd51300a8d1847f7fd73df372c49464bafe09703eb8774e8e2d8e4"
Sep 16 01:22:35.309260 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:22:35.309241    2573 scope.go:117] "RemoveContainer" containerID="a6a484cd541a783859c2f0d546d472a84d1b8c007b3a89120c6ca06a63a63a1f"
Sep 16 01:23:02.664316 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664265    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"]
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664717    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c" containerName="manager"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664731    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c" containerName="manager"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664750    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23708c05-b8e7-41e1-aa1e-d8046892d4b0" containerName="revoke-keys"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664755    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="23708c05-b8e7-41e1-aa1e-d8046892d4b0" containerName="revoke-keys"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664763    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664769    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664779    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664785    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664853    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664861    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664867    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="23708c05-b8e7-41e1-aa1e-d8046892d4b0" containerName="revoke-keys"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664875    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy"
Sep 16 01:23:02.665004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.664882    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6a2e87-2da9-4d7e-83ce-2ac266f77e5c" containerName="manager"
Sep 16 01:23:02.668277 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.668255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:02.670690 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.670658    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 01:23:02.670839 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.670691    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 01:23:02.670839 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.670695    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wsn7n\""
Sep 16 01:23:02.681005 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.680980    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"]
Sep 16 01:23:02.791977 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.791945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrndf\" (UniqueName: \"kubernetes.io/projected/827cff0f-8297-4a56-bcd2-878958ac7b7e-kube-api-access-nrndf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jh4xs\" (UID: \"827cff0f-8297-4a56-bcd2-878958ac7b7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:02.792143 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.792011    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/827cff0f-8297-4a56-bcd2-878958ac7b7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jh4xs\" (UID: \"827cff0f-8297-4a56-bcd2-878958ac7b7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:02.893303 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.893266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/827cff0f-8297-4a56-bcd2-878958ac7b7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jh4xs\" (UID: \"827cff0f-8297-4a56-bcd2-878958ac7b7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:02.893472 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.893371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrndf\" (UniqueName: \"kubernetes.io/projected/827cff0f-8297-4a56-bcd2-878958ac7b7e-kube-api-access-nrndf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jh4xs\" (UID: \"827cff0f-8297-4a56-bcd2-878958ac7b7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:02.893782 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.893761    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/827cff0f-8297-4a56-bcd2-878958ac7b7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jh4xs\" (UID: \"827cff0f-8297-4a56-bcd2-878958ac7b7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:02.902332 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.902311    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrndf\" (UniqueName: \"kubernetes.io/projected/827cff0f-8297-4a56-bcd2-878958ac7b7e-kube-api-access-nrndf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jh4xs\" (UID: \"827cff0f-8297-4a56-bcd2-878958ac7b7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:02.982148 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:02.982115    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:03.263753 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:03.263724    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"]
Sep 16 01:23:03.265306 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:23:03.265267    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827cff0f_8297_4a56_bcd2_878958ac7b7e.slice/crio-e18abc98b2bec92e3fe80b575094b1d3a49ebf8f40b8cb0e4e1fd252607a0736 WatchSource:0}: Error finding container e18abc98b2bec92e3fe80b575094b1d3a49ebf8f40b8cb0e4e1fd252607a0736: Status 404 returned error can't find the container with id e18abc98b2bec92e3fe80b575094b1d3a49ebf8f40b8cb0e4e1fd252607a0736
Sep 16 01:23:03.680971 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:03.680883    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs" event={"ID":"827cff0f-8297-4a56-bcd2-878958ac7b7e","Type":"ContainerStarted","Data":"36e617cd538d7cd772cd8f7023240aaeaf94967164dcae6d58e1f359ecc247ce"}
Sep 16 01:23:03.680971 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:03.680950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs" event={"ID":"827cff0f-8297-4a56-bcd2-878958ac7b7e","Type":"ContainerStarted","Data":"e18abc98b2bec92e3fe80b575094b1d3a49ebf8f40b8cb0e4e1fd252607a0736"}
Sep 16 01:23:03.681365 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:03.680989    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:03.702238 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:03.702191    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs" podStartSLOduration=1.702175167 podStartE2EDuration="1.702175167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:03.700329559 +0000 UTC m=+2369.403151750" watchObservedRunningTime="2026-09-16 01:23:03.702175167 +0000 UTC m=+2369.404997359"
Sep 16 01:23:09.765581 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:09.765548    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:23:14.689004 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:14.688974    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jh4xs"
Sep 16 01:23:23.765301 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:23.765269    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:23:35.339781 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:35.339746    2573 scope.go:117] "RemoveContainer" containerID="88ea7a679883cbe7323a9b9ec8f909f3298a9fe36006c1d12f6f133f273a6a07"
Sep 16 01:23:35.349103 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:35.349083    2573 scope.go:117] "RemoveContainer" containerID="7422a28b8619b3506f54ae5d68b0080ea27491bde2504a6c229d24c0b4095d14"
Sep 16 01:23:35.356921 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:35.356899    2573 scope.go:117] "RemoveContainer" containerID="fb74d79a6c4b12e12bdd99cbdd1ba893b0c84cb827edd88f2d1663346f9b2023"
Sep 16 01:23:56.381147 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.381060    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"]
Sep 16 01:23:56.381594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.381439    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy"
Sep 16 01:23:56.381594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.381450    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8459a3f-245f-42bd-ba92-29fc424c65fa" containerName="istio-proxy"
Sep 16 01:23:56.381594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.381478    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy"
Sep 16 01:23:56.381594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.381484    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy"
Sep 16 01:23:56.381594 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.381569    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f554aabb-aac0-4371-9934-1cdf2ea7d6ba" containerName="istio-proxy"
Sep 16 01:23:56.384851 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.384834    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.386932 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.386912    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d04d108c-kserve-self-signed-certs\""
Sep 16 01:23:56.394622 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.394602    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"]
Sep 16 01:23:56.456159 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.456131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-dshm\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.456309 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.456169    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2mb4\" (UniqueName: \"kubernetes.io/projected/cab515d3-faa2-4a19-864c-bf7f4a0342bc-kube-api-access-b2mb4\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.456309 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.456196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-model-cache\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.456309 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.456270    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-home\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.456467 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.456333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tls-certs\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.456467 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.456371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tmp-dir\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.557477 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.557444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-dshm\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.557650 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.557482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2mb4\" (UniqueName: \"kubernetes.io/projected/cab515d3-faa2-4a19-864c-bf7f4a0342bc-kube-api-access-b2mb4\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.557650 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.557505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-model-cache\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.557650 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.557555    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-home\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.557650 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.557581    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tls-certs\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.557650 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.557608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tmp-dir\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.557996 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.557978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-model-cache\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.558059 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.558025    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-home\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.558059 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.558036    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tmp-dir\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.559915 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.559895    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-dshm\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.560328 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.560304    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tls-certs\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.565055 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.565032    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2mb4\" (UniqueName: \"kubernetes.io/projected/cab515d3-faa2-4a19-864c-bf7f4a0342bc-kube-api-access-b2mb4\") pod \"e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") " pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.694559 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.694470    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:23:56.970493 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:56.970470    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"]
Sep 16 01:23:56.973499 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:23:56.973471    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcab515d3_faa2_4a19_864c_bf7f4a0342bc.slice/crio-f4ced7e4d7c00e7fb2da32854f8ce898e835229a2a24b92d9fecc83ac1daed64 WatchSource:0}: Error finding container f4ced7e4d7c00e7fb2da32854f8ce898e835229a2a24b92d9fecc83ac1daed64: Status 404 returned error can't find the container with id f4ced7e4d7c00e7fb2da32854f8ce898e835229a2a24b92d9fecc83ac1daed64
Sep 16 01:23:57.970473 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:57.970436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn" event={"ID":"cab515d3-faa2-4a19-864c-bf7f4a0342bc","Type":"ContainerStarted","Data":"89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b"}
Sep 16 01:23:57.970473 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:57.970474    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn" event={"ID":"cab515d3-faa2-4a19-864c-bf7f4a0342bc","Type":"ContainerStarted","Data":"f4ced7e4d7c00e7fb2da32854f8ce898e835229a2a24b92d9fecc83ac1daed64"}
Sep 16 01:23:57.987595 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:23:57.987544    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn" podStartSLOduration=1.987504402 podStartE2EDuration="1.987504402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:57.984693773 +0000 UTC m=+2423.687515975" watchObservedRunningTime="2026-09-16 01:23:57.987504402 +0000 UTC m=+2423.690326597"
Sep 16 01:24:06.694834 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:06.694796    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:24:06.694834 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:06.694843    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:24:06.707849 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:06.707825    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:24:07.033823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:07.033795    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:24:15.765136 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:15.765109    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:24:35.383914 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:35.383878    2573 scope.go:117] "RemoveContainer" containerID="ab56ecfdd99b6606f456af9b576eb2f3ed876635e7e80a0cd8b0b0940d6f2468"
Sep 16 01:24:40.699165 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:40.699129    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"]
Sep 16 01:24:51.765111 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:51.765081    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:24:55.722043 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:55.722006    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn" podUID="cab515d3-faa2-4a19-864c-bf7f4a0342bc" containerName="main" containerID="cri-o://89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b" gracePeriod=45
Sep 16 01:24:55.977398 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:55.977343    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:24:56.085962 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.085935    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tmp-dir\") pod \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") "
Sep 16 01:24:56.086133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.085979    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-home\") pod \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") "
Sep 16 01:24:56.086133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086004    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2mb4\" (UniqueName: \"kubernetes.io/projected/cab515d3-faa2-4a19-864c-bf7f4a0342bc-kube-api-access-b2mb4\") pod \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") "
Sep 16 01:24:56.086133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086019    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-model-cache\") pod \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") "
Sep 16 01:24:56.086133 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086121    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tls-certs\") pod \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") "
Sep 16 01:24:56.086329 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086160    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-dshm\") pod \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\" (UID: \"cab515d3-faa2-4a19-864c-bf7f4a0342bc\") "
Sep 16 01:24:56.086329 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086261    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-home" (OuterVolumeSpecName: "home") pod "cab515d3-faa2-4a19-864c-bf7f4a0342bc" (UID: "cab515d3-faa2-4a19-864c-bf7f4a0342bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:24:56.086329 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086265    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cab515d3-faa2-4a19-864c-bf7f4a0342bc" (UID: "cab515d3-faa2-4a19-864c-bf7f4a0342bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:24:56.086490 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086360    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "cab515d3-faa2-4a19-864c-bf7f4a0342bc" (UID: "cab515d3-faa2-4a19-864c-bf7f4a0342bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:24:56.086490 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086428    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:24:56.086490 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086440    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:24:56.086490 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.086448    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:24:56.088403 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.088375    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab515d3-faa2-4a19-864c-bf7f4a0342bc-kube-api-access-b2mb4" (OuterVolumeSpecName: "kube-api-access-b2mb4") pod "cab515d3-faa2-4a19-864c-bf7f4a0342bc" (UID: "cab515d3-faa2-4a19-864c-bf7f4a0342bc"). InnerVolumeSpecName "kube-api-access-b2mb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:24:56.088403 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.088396    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-dshm" (OuterVolumeSpecName: "dshm") pod "cab515d3-faa2-4a19-864c-bf7f4a0342bc" (UID: "cab515d3-faa2-4a19-864c-bf7f4a0342bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:24:56.088578 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.088405    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cab515d3-faa2-4a19-864c-bf7f4a0342bc" (UID: "cab515d3-faa2-4a19-864c-bf7f4a0342bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:24:56.187188 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.187162    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2mb4\" (UniqueName: \"kubernetes.io/projected/cab515d3-faa2-4a19-864c-bf7f4a0342bc-kube-api-access-b2mb4\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:24:56.187188 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.187187    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab515d3-faa2-4a19-864c-bf7f4a0342bc-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:24:56.187346 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.187196    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab515d3-faa2-4a19-864c-bf7f4a0342bc-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Sep 16 01:24:56.276116 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.276053    2573 generic.go:356] "Generic (PLEG): container finished" podID="cab515d3-faa2-4a19-864c-bf7f4a0342bc" containerID="89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b" exitCode=0
Sep 16 01:24:56.276116 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.276095    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"
Sep 16 01:24:56.276116 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.276108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn" event={"ID":"cab515d3-faa2-4a19-864c-bf7f4a0342bc","Type":"ContainerDied","Data":"89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b"}
Sep 16 01:24:56.276306 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.276139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn" event={"ID":"cab515d3-faa2-4a19-864c-bf7f4a0342bc","Type":"ContainerDied","Data":"f4ced7e4d7c00e7fb2da32854f8ce898e835229a2a24b92d9fecc83ac1daed64"}
Sep 16 01:24:56.276306 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.276156    2573 scope.go:117] "RemoveContainer" containerID="89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b"
Sep 16 01:24:56.285496 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.285477    2573 scope.go:117] "RemoveContainer" containerID="89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b"
Sep 16 01:24:56.285812 ip-10-0-131-89 kubenswrapper[2573]: E0916 01:24:56.285784    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b\": container with ID starting with 89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b not found: ID does not exist" containerID="89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b"
Sep 16 01:24:56.285871 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.285821    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b"} err="failed to get container status \"89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b\": rpc error: code = NotFound desc = could not find container \"89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b\": container with ID starting with 89eb3e9086749f4bcee5d3abf482cfd006123a876b892b13064dd0aed68bc17b not found: ID does not exist"
Sep 16 01:24:56.296596 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.296574    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"]
Sep 16 01:24:56.299090 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.299071    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d04d108c-kserve-7d5498fc6c-mwbcn"]
Sep 16 01:24:56.772049 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:24:56.772022    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab515d3-faa2-4a19-864c-bf7f4a0342bc" path="/var/lib/kubelet/pods/cab515d3-faa2-4a19-864c-bf7f4a0342bc/volumes"
Sep 16 01:25:35.406041 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:25:35.405963    2573 scope.go:117] "RemoveContainer" containerID="95fa5909b99eb255b08a98ebe6ec181455fb19b154293b838cf176e423f89c07"
Sep 16 01:25:35.416485 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:25:35.416462    2573 scope.go:117] "RemoveContainer" containerID="846f8ab13ef575c0dcf8e7756d88391d8ff0dc362173b69158a5478dc875aab7"
Sep 16 01:25:36.765282 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:25:36.765249    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:26:14.767024 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:26:14.766995    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:26:41.764580 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:26:41.764543    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:27:29.764717 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:27:29.764681    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:27:46.764735 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:27:46.764704    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-gr57w" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:28:33.076651 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:33.076560    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qt698_e0721d84-496f-43ef-876c-8be3c9dc88ee/global-pull-secret-syncer/0.log"
Sep 16 01:28:33.211858 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:33.211822    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rnb7l_b21bf3a4-5aae-4634-b834-08b8e8629924/konnectivity-agent/0.log"
Sep 16 01:28:33.307169 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:33.307138    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-89.ec2.internal_6b56cb1fea5efc140deb885904593735/haproxy/0.log"
Sep 16 01:28:38.213025 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:38.212991    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-jh4xs_827cff0f-8297-4a56-bcd2-878958ac7b7e/manager/0.log"
Sep 16 01:28:40.012948 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.012904    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-4r5r7_226d6ebd-5397-4939-b861-33f1d5da4a10/cluster-monitoring-operator/0.log"
Sep 16 01:28:40.111071 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.111000    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6696656695-2x5tk_78375bf6-9446-43bd-8027-77162a7315d0/metrics-server/0.log"
Sep 16 01:28:40.307640 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.307614    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6jj7_c884b58b-cf9d-43d6-a79d-2f25cbebd729/node-exporter/0.log"
Sep 16 01:28:40.323359 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.323338    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6jj7_c884b58b-cf9d-43d6-a79d-2f25cbebd729/kube-rbac-proxy/0.log"
Sep 16 01:28:40.342130 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.342107    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6jj7_c884b58b-cf9d-43d6-a79d-2f25cbebd729/init-textfile/0.log"
Sep 16 01:28:40.543358 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.543335    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c201f19-8f74-42e7-9367-18d2fc0045fb/prometheus/0.log"
Sep 16 01:28:40.565776 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.565754    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c201f19-8f74-42e7-9367-18d2fc0045fb/config-reloader/0.log"
Sep 16 01:28:40.585921 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.585901    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c201f19-8f74-42e7-9367-18d2fc0045fb/thanos-sidecar/0.log"
Sep 16 01:28:40.607304 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.607283    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c201f19-8f74-42e7-9367-18d2fc0045fb/kube-rbac-proxy-web/0.log"
Sep 16 01:28:40.627817 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.627795    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c201f19-8f74-42e7-9367-18d2fc0045fb/kube-rbac-proxy/0.log"
Sep 16 01:28:40.646826 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.646809    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c201f19-8f74-42e7-9367-18d2fc0045fb/kube-rbac-proxy-thanos/0.log"
Sep 16 01:28:40.667896 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.667844    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c201f19-8f74-42e7-9367-18d2fc0045fb/init-config-reloader/0.log"
Sep 16 01:28:40.743625 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.743601    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-cn7nr_49971735-36c4-47ad-b8f9-2afd41dd2015/prometheus-operator-admission-webhook/0.log"
Sep 16 01:28:40.846754 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.846724    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78b986c49b-ptn4w_75ee21fc-48e2-48c7-a47d-6fafbe636418/thanos-query/0.log"
Sep 16 01:28:40.866719 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.866701    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78b986c49b-ptn4w_75ee21fc-48e2-48c7-a47d-6fafbe636418/kube-rbac-proxy-web/0.log"
Sep 16 01:28:40.889655 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.889629    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78b986c49b-ptn4w_75ee21fc-48e2-48c7-a47d-6fafbe636418/kube-rbac-proxy/0.log"
Sep 16 01:28:40.925477 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.925426    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78b986c49b-ptn4w_75ee21fc-48e2-48c7-a47d-6fafbe636418/prom-label-proxy/0.log"
Sep 16 01:28:40.983834 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:40.983810    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78b986c49b-ptn4w_75ee21fc-48e2-48c7-a47d-6fafbe636418/kube-rbac-proxy-rules/0.log"
Sep 16 01:28:41.037823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:41.037782    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78b986c49b-ptn4w_75ee21fc-48e2-48c7-a47d-6fafbe636418/kube-rbac-proxy-metrics/0.log"
Sep 16 01:28:41.764823 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:41.764791    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-87r96" secret="" err="secret \"payload-processing-dockercfg-2xnr2\" not found"
Sep 16 01:28:43.216663 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.216633    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"]
Sep 16 01:28:43.217083 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.217061    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab515d3-faa2-4a19-864c-bf7f4a0342bc" containerName="main"
Sep 16 01:28:43.217083 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.217082    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab515d3-faa2-4a19-864c-bf7f4a0342bc" containerName="main"
Sep 16 01:28:43.217289 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.217182    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab515d3-faa2-4a19-864c-bf7f4a0342bc" containerName="main"
Sep 16 01:28:43.220152 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.220137    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.222261 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.222241    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xll2x\"/\"kube-root-ca.crt\""
Sep 16 01:28:43.222396 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.222287    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xll2x\"/\"default-dockercfg-fdxcj\""
Sep 16 01:28:43.223062 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.223044    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xll2x\"/\"openshift-service-ca.crt\""
Sep 16 01:28:43.227694 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.227267    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"]
Sep 16 01:28:43.366608 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.366576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-proc\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.366761 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.366631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-sys\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.366761 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.366695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-lib-modules\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.366761 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.366717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b55pz\" (UniqueName: \"kubernetes.io/projected/9716a607-1136-4378-affd-c8d4d3620520-kube-api-access-b55pz\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.366761 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.366750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-podres\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.467745 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-proc\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.467745 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-sys\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.467950 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467760    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-lib-modules\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.467950 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b55pz\" (UniqueName: \"kubernetes.io/projected/9716a607-1136-4378-affd-c8d4d3620520-kube-api-access-b55pz\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.467950 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-podres\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.467950 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467799    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-proc\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.467950 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467898    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-sys\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.467950 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467909    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-podres\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.468159 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.467923    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9716a607-1136-4378-affd-c8d4d3620520-lib-modules\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.475020 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.474998    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b55pz\" (UniqueName: \"kubernetes.io/projected/9716a607-1136-4378-affd-c8d4d3620520-kube-api-access-b55pz\") pod \"perf-node-gather-daemonset-9chxs\" (UID: \"9716a607-1136-4378-affd-c8d4d3620520\") " pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.531553 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.531511    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:43.807212 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:43.807189    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"]
Sep 16 01:28:43.809727 ip-10-0-131-89 kubenswrapper[2573]: W0916 01:28:43.809693    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9716a607_1136_4378_affd_c8d4d3620520.slice/crio-048caf2d54bd4abd8431fe5113cc88279a74cc7af26a16664d5155c3e93aad16 WatchSource:0}: Error finding container 048caf2d54bd4abd8431fe5113cc88279a74cc7af26a16664d5155c3e93aad16: Status 404 returned error can't find the container with id 048caf2d54bd4abd8431fe5113cc88279a74cc7af26a16664d5155c3e93aad16
Sep 16 01:28:44.353677 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:44.353639    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-pvhr4_eee257f5-5e28-4785-ab61-b9d403220ec4/volume-data-source-validator/0.log"
Sep 16 01:28:44.414020 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:44.413993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs" event={"ID":"9716a607-1136-4378-affd-c8d4d3620520","Type":"ContainerStarted","Data":"47713ee965ce9d665c6658938326a6405e4491515113437684526bc55d4b5074"}
Sep 16 01:28:44.414166 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:44.414046    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs" event={"ID":"9716a607-1136-4378-affd-c8d4d3620520","Type":"ContainerStarted","Data":"048caf2d54bd4abd8431fe5113cc88279a74cc7af26a16664d5155c3e93aad16"}
Sep 16 01:28:44.414166 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:44.414107    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:44.428939 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:44.428888    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs" podStartSLOduration=1.428875591 podStartE2EDuration="1.428875591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:28:44.427133108 +0000 UTC m=+2710.129955306" watchObservedRunningTime="2026-09-16 01:28:44.428875591 +0000 UTC m=+2710.131697786"
Sep 16 01:28:45.462450 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:45.462419    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7m8gh_59b8ae0f-6819-4e75-8d12-27c5156bcd9d/dns/0.log"
Sep 16 01:28:45.481810 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:45.481788    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7m8gh_59b8ae0f-6819-4e75-8d12-27c5156bcd9d/kube-rbac-proxy/0.log"
Sep 16 01:28:45.637573 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:45.637540    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rhklt_93bd702c-41a7-4c5b-b7a2-24428b0e7a18/dns-node-resolver/0.log"
Sep 16 01:28:46.139189 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:46.139163    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68596fcf69-rkdfq_9bc456b8-82e4-49ce-9b21-bd54571af7dd/registry/0.log"
Sep 16 01:28:46.181585 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:46.181561    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ch47b_0a9eb6d9-33b8-4ded-b33e-5f1facefbf9e/node-ca/0.log"
Sep 16 01:28:47.175924 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:47.175879    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c92k86_4fa70455-2958-412f-85db-05e6c74c80d0/istio-proxy/0.log"
Sep 16 01:28:47.569029 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:47.568989    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-kdnf6_2f9baea0-1171-4f4e-a686-75ad536a98e7/istio-proxy/0.log"
Sep 16 01:28:47.573627 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:47.573598    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-kdnf6_2f9baea0-1171-4f4e-a686-75ad536a98e7/istio-proxy/1.log"
Sep 16 01:28:47.590779 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:47.590754    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-gr57w_bb2333dd-f5e0-4c73-af19-22ae153024c8/payload-pre-processing/0.log"
Sep 16 01:28:47.611969 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:47.611946    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-87r96_b3056300-83f7-42c9-97ac-82fff85a22b8/payload-processing/0.log"
Sep 16 01:28:47.642057 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:47.642032    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74867654d6-ccpxx_9ef9603f-db2f-47c9-a78a-eec9b7f52d94/router/0.log"
Sep 16 01:28:48.500076 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:48.500043    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hf8lr_e064abea-ce8c-4fad-b012-d23a5fbf95ab/serve-healthcheck-canary/0.log"
Sep 16 01:28:48.981432 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:48.981403    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5dcs9_66b2ae7e-9911-45c4-8247-b5b4b55d83b6/kube-rbac-proxy/0.log"
Sep 16 01:28:49.000083 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:49.000049    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5dcs9_66b2ae7e-9911-45c4-8247-b5b4b55d83b6/exporter/0.log"
Sep 16 01:28:49.021457 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:49.021436    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5dcs9_66b2ae7e-9911-45c4-8247-b5b4b55d83b6/extractor/0.log"
Sep 16 01:28:50.428449 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:50.428416    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xll2x/perf-node-gather-daemonset-9chxs"
Sep 16 01:28:51.200069 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:51.200043    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-689bfb855b-xbd27_6641e296-c644-4a27-ae3c-b2bc4df877b0/manager/0.log"
Sep 16 01:28:51.218554 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:51.218515    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9zc66_c8a716f3-27b4-40ce-9a6a-d1ef0b3bd2c1/manager/0.log"
Sep 16 01:28:51.621971 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:51.621890    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6gvb5_1497812b-fa7c-4aa9-9270-7d60bb287164/manager/0.log"
Sep 16 01:28:51.685214 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:51.685190    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-t7zkb_695cf87c-0087-451d-afba-cbcf5b67bff4/server/0.log"
Sep 16 01:28:51.816383 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:51.816358    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kgj5q_18c757f2-5e60-4944-bd69-b1ae2ae3b647/manager/0.log"
Sep 16 01:28:51.838909 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:51.838882    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9778fc5bd-hnmlh_f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c/manager/0.log"
Sep 16 01:28:51.857040 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:51.857021    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9778fc5bd-hnmlh_f7fc9abd-63d9-47a4-a8b6-bdfbf76b6c6c/copy-manifests/0.log"
Sep 16 01:28:53.563988 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:53.563901    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54b79494c6-4h7nt_2dbca285-f758-413f-929b-280070f09d27/manager/0.log"
Sep 16 01:28:54.070586 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:54.070555    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c92k86_4fa70455-2958-412f-85db-05e6c74c80d0/istio-proxy/0.log"
Sep 16 01:28:54.441271 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:54.441161    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-kdnf6_2f9baea0-1171-4f4e-a686-75ad536a98e7/istio-proxy/0.log"
Sep 16 01:28:54.445618 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:54.445585    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-kdnf6_2f9baea0-1171-4f4e-a686-75ad536a98e7/istio-proxy/1.log"
Sep 16 01:28:54.455283 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:54.455255    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-gr57w_bb2333dd-f5e0-4c73-af19-22ae153024c8/payload-pre-processing/0.log"
Sep 16 01:28:54.470012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:54.469995    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-87r96_b3056300-83f7-42c9-97ac-82fff85a22b8/payload-processing/0.log"
Sep 16 01:28:54.489092 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:54.489072    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74867654d6-ccpxx_9ef9603f-db2f-47c9-a78a-eec9b7f52d94/router/0.log"
Sep 16 01:28:55.950750 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:55.950714    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c92k86_4fa70455-2958-412f-85db-05e6c74c80d0/istio-proxy/0.log"
Sep 16 01:28:56.322771 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:56.322661    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-kdnf6_2f9baea0-1171-4f4e-a686-75ad536a98e7/istio-proxy/0.log"
Sep 16 01:28:56.327280 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:56.327260    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-kdnf6_2f9baea0-1171-4f4e-a686-75ad536a98e7/istio-proxy/1.log"
Sep 16 01:28:56.340314 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:56.340297    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-gr57w_bb2333dd-f5e0-4c73-af19-22ae153024c8/payload-pre-processing/0.log"
Sep 16 01:28:56.356279 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:56.356257    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-87r96_b3056300-83f7-42c9-97ac-82fff85a22b8/payload-processing/0.log"
Sep 16 01:28:56.384362 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:56.384343    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74867654d6-ccpxx_9ef9603f-db2f-47c9-a78a-eec9b7f52d94/router/0.log"
Sep 16 01:28:58.940544 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:58.940485    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9982t_99772ee8-b1af-4d1c-acd4-55ea52a1a114/kube-storage-version-migrator-operator/1.log"
Sep 16 01:28:58.942199 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:28:58.942168    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9982t_99772ee8-b1af-4d1c-acd4-55ea52a1a114/kube-storage-version-migrator-operator/0.log"
Sep 16 01:29:00.068685 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.068652    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rp8ph_12358ed4-8f49-4749-bb23-62f5afd0a8c2/kube-multus-additional-cni-plugins/0.log"
Sep 16 01:29:00.088983 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.088952    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rp8ph_12358ed4-8f49-4749-bb23-62f5afd0a8c2/egress-router-binary-copy/0.log"
Sep 16 01:29:00.108400 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.108370    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rp8ph_12358ed4-8f49-4749-bb23-62f5afd0a8c2/cni-plugins/0.log"
Sep 16 01:29:00.127012 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.126990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rp8ph_12358ed4-8f49-4749-bb23-62f5afd0a8c2/bond-cni-plugin/0.log"
Sep 16 01:29:00.145587 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.145565    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rp8ph_12358ed4-8f49-4749-bb23-62f5afd0a8c2/routeoverride-cni/0.log"
Sep 16 01:29:00.164265 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.164242    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rp8ph_12358ed4-8f49-4749-bb23-62f5afd0a8c2/whereabouts-cni-bincopy/0.log"
Sep 16 01:29:00.182448 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.182433    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rp8ph_12358ed4-8f49-4749-bb23-62f5afd0a8c2/whereabouts-cni/0.log"
Sep 16 01:29:00.432015 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.431933    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qtzkc_ca0e895f-b157-4c49-884c-0bc90b74e724/kube-multus/0.log"
Sep 16 01:29:00.578127 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.578096    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nbb66_5527c19f-cd6a-4cd6-adc4-0bb7e70f140d/network-metrics-daemon/0.log"
Sep 16 01:29:00.591543 ip-10-0-131-89 kubenswrapper[2573]: I0916 01:29:00.591492    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nbb66_5527c19f-cd6a-4cd6-adc4-0bb7e70f140d/kube-rbac-proxy/0.log"