Sep 16 13:08:48.457187 ip-10-0-134-164 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 13:08:48.876861 ip-10-0-134-164 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:08:48.876861 ip-10-0-134-164 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 13:08:48.876861 ip-10-0-134-164 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:08:48.876861 ip-10-0-134-164 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 13:08:48.876861 ip-10-0-134-164 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:08:48.877537 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.876974    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 13:08:48.880094 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880077    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:08:48.880094 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880093    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880096    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880100    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880103    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880106    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880109    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880112    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880114    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880117    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880119    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880122    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880125    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880127    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880131    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880133    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880136    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880138    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880142    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880147    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880149    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:08:48.880155 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880152    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880155    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880157    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880160    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880163    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880166    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880170    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880174    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880176    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880179    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880181    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880184    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880187    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880191    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880194    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880197    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880200    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880208    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880211    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:08:48.880637 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880214    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880217    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880220    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880222    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880225    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880227    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880229    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880232    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880234    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880237    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880239    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880241    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880244    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880246    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880248    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880252    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880254    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880257    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880259    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880262    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:08:48.881115 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880264    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880267    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880269    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880272    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880274    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880276    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880279    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880281    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880284    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880287    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880291    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880300    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880304    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880307    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880309    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880312    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880314    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880317    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880320    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880322    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:08:48.881604 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880325    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880327    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880330    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880332    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880335    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880338    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880341    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880344    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880347    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880349    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880352    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880354    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880357    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880359    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880362    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880365    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880367    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880370    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.880372    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:08:48.882097 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881137    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881146    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881149    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881152    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881154    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881158    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881160    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881163    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881167    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881171    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881174    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881176    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881179    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881181    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881184    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881186    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881189    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881192    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881194    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881197    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:08:48.882560 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881201    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881203    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881206    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881208    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881211    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881214    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881216    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881219    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881221    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881224    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881227    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881231    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881234    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881237    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881239    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881242    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881244    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881247    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881249    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881252    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:08:48.883079 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881255    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881257    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881260    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881262    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881265    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881267    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881270    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881273    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881275    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881279    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881282    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881284    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881287    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881290    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881292    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881295    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881297    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881300    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881303    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:08:48.883653 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881305    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881307    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881310    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881313    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881316    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881318    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881321    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881323    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881326    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881329    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881331    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881333    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881336    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881339    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881342    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881344    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881347    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881349    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881351    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881354    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:08:48.884132 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881356    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881359    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881361    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881364    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881367    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881370    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881373    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881375    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881378    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881381    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881383    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881386    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881388    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881391    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881393    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881396    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881398    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881401    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881404    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881406    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:08:48.884610 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881771    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881776    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881779    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881781    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881784    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881787    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881791    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881794    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881796    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881799    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881802    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881804    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881807    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881810    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881812    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881815    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881818    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881820    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881823    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881825    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:08:48.885124 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881828    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881831    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881833    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881836    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881838    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881841    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881844    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881846    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881864    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881867    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881870    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881872    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881876    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881879    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881883    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881886    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881889    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881893    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881895    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:08:48.885639 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881898    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881901    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881905    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881908    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881911    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881914    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881919    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881921    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881924    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881927    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881929    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881932    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881934    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881937    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881939    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881942    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881945    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881948    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881951    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881953    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:08:48.886125 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881956    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881958    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881960    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881963    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881965    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881968    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881970    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881974    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881977    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881980    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881982    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881985    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881988    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881990    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881993    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881995    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.881998    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882000    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882004    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882007    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:08:48.886609 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882009    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882012    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882014    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882017    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882019    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882022    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882024    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882027    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882029    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882032    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882034    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882037    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882039    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882042    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882044    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882047    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882050    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882054    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882056    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:08:48.887163 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882061    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882149    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882192    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882200    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882205    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882209    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882213    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882218    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882222    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882225    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882228    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882233    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882237    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882240    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882243    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882246    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882249    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882252    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882255    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882259    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882262    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882265    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882268    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882271    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 13:08:48.887616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882275    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882278    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882281    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882284    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882288    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882291    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882294    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882297    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882300    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882305    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882308    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882310    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882313    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882316    2577 flags.go:64] FLAG: --enable-server="true"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882320    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882324    2577 flags.go:64] FLAG: --event-burst="100"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882327    2577 flags.go:64] FLAG: --event-qps="50"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882330    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882333    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882338    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882341    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882344    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882347    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882350    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882353    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 13:08:48.888215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882356    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882359    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882362    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882365    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882368    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882370    2577 flags.go:64] FLAG: --feature-gates=""
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882374    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882377    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882380    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882383    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882386    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882389    2577 flags.go:64] FLAG: --help="false"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882392    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-164.ec2.internal"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882395    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882398    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882401    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882404    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882408    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882410    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882413    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882416    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882419    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882422    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882426    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 13:08:48.888877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882428    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882432    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882436    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882439    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882441    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882444    2577 flags.go:64] FLAG: --lock-file=""
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882447    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882450    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882453    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882458    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882460    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882463    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882466    2577 flags.go:64] FLAG: --logging-format="text"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882469    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882473    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882476    2577 flags.go:64] FLAG: --manifest-url=""
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882478    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882488    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882491    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882499    2577 flags.go:64] FLAG: --max-pods="110"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882502    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882505    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882508    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882511    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882514    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 13:08:48.889462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882517    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882520    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882527    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882530    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882533    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882536    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882539    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882545    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882548    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882551    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882555    2577 flags.go:64] FLAG: --port="10250"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882558    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882561    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0370b9b9f0f9464e4"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882564    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882568    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882571    2577 flags.go:64] FLAG: --register-node="true"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882574    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882578    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882581    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882583    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882586    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882590    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882593    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882596    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882599    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 13:08:48.890095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882601    2577 flags.go:64] FLAG: --runonce="false"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882604    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882607    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882610    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882613    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882616    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882619    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882622    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882625    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882628    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882630    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882633    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882636    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882639    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882642    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882645    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882650    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882654    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882656    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882660    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882663    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882665    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882668    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882671    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882673    2577 flags.go:64] FLAG: --v="2"
Sep 16 13:08:48.890707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882677    2577 flags.go:64] FLAG: --version="false"
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882681    2577 flags.go:64] FLAG: --vmodule=""
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882687    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.882690    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882804    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882808    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882811    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882813    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882816    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882819    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882821    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882824    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882826    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882829    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882831    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882834    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882837    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882839    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882842    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882844    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882862    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:08:48.891335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882865    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882868    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882870    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882873    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882876    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882879    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882882    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882884    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882887    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882890    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882893    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882895    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882898    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882902    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882905    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882908    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882910    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882913    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882916    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882919    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:08:48.891865 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882921    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882924    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882926    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882929    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882931    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882934    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882937    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882939    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882941    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882945    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882949    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882952    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882955    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882958    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882962    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882964    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882967    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882969    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882972    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:08:48.892405 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882974    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882977    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882980    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882982    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882985    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882987    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882991    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882994    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882996    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.882999    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883001    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883005    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883008    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883012    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883016    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883019    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883021    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883024    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883027    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:08:48.892885 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883030    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883032    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883035    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883037    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883040    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883043    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883045    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883048    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883050    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883054    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883056    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883059    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883061    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883064    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883066    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883069    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883071    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883074    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883076    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883081    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:08:48.893357 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883084    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883087    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883089    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.883092    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.883796    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.891646    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.891663    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.891678    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.891686    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891747    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891751    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891754    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891757    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891760    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:08:48.893946 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891763    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891766    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891769    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891772    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891774    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891777    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891780    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891783    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891786    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891789    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891791    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891794    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891797    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891800    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891802    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891805    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891807    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891811    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891813    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:08:48.894567 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891816    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891819    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891822    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891824    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891827    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891835    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891840    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891843    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891846    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891863    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891867    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891870    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891874    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891876    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891879    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891882    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891886    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891890    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891893    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:08:48.895218 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891895    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891898    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891901    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891904    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891906    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891909    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891911    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891914    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891917    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891919    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891922    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891924    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891927    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891929    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891932    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891934    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891937    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891940    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891942    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891951    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:08:48.895683 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891954    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891956    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891959    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891961    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891964    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891967    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891969    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891971    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891974    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891976    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891979    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891982    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891986    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891989    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891992    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891995    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.891999    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892002    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892004    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:08:48.896224 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892007    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892009    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892012    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892015    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892017    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892020    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892022    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892025    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892027    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892030    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892032    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892035    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892037    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892045    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892048    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892050    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:08:48.896681 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892053    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.892058    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892205    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892210    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892213    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892215    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892218    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892220    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892223    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892225    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892228    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892231    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892234    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892237    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892239    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:08:48.897106 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892242    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892244    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892247    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892249    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892252    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892255    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892257    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892259    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892262    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892264    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892267    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892269    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892272    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892274    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892277    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892285    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892287    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892290    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892292    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892295    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:08:48.897486 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892298    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892300    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892303    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892305    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892308    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892311    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892313    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892316    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892318    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892321    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892324    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892326    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892329    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892331    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892334    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892336    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892340    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892343    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892345    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892348    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:08:48.897989 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892350    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892353    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892355    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892358    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892360    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892363    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892365    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892368    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892376    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892378    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892381    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892383    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892386    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892388    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892391    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892393    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892396    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892402    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892404    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:08:48.898481 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892407    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892411    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892415    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892418    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892421    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892424    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892426    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892429    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892432    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892434    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892437    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892439    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892442    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892444    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892447    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892449    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892452    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892454    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892457    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:08:48.898949 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892459    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892462    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892465    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892477    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892480    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892484    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892487    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:48.892490    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.892495    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.892624    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 13:08:48.899413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.898277    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 13:08:48.899662 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.899653    2577 server.go:1019] "Starting client certificate rotation"
Sep 16 13:08:48.899768 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.899749    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 13:08:48.899827 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.899817    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 13:08:48.902501 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.902489    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 13:08:48.903597 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.903579    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 13:08:48.914807 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.914783    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 16 13:08:48.917906 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.917892    2577 log.go:25] "Validated CRI v1 image API"
Sep 16 13:08:48.918718 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.918699    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 13:08:48.921955 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.921929    2577 fs.go:135] Filesystem UUIDs: map[670fab2a-1356-4a8d-9d9c-0c10f01a3767:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bf8dd3d0-9cda-4d21-9aeb-ae097f69dbdf:/dev/nvme0n1p3]
Sep 16 13:08:48.922025 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.921953    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 13:08:48.927727 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.927621    2577 manager.go:216] Machine: {Timestamp:2026-09-16 13:08:48.927148288 +0000 UTC m=+0.349889859 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100434 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2756ab482bb49de4eb4db87c8b4113 SystemUUID:ec2756ab-482b-b49d-e4eb-4db87c8b4113 BootID:cdebaac0-b36a-4c6f-ad4d-0a06eefbf399 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:11:25:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:11:25:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:11:57:74:ee:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 13:08:48.927727 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.927725    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 13:08:48.927832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.927804    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 13:08:48.931048 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.931032    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 13:08:48.931310 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.931283    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 13:08:48.931463 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.931315    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-164.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 13:08:48.932151 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.932141    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 13:08:48.932188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.932153    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 13:08:48.932188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.932162    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 13:08:48.933079 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.933069    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 13:08:48.934027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.934016    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 13:08:48.937406 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.937395    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 13:08:48.937520 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.937511    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 13:08:48.937555 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.937540    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 13:08:48.937582 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.937568    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 13:08:48.937612 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.937600    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 13:08:48.937664 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.937655    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 16 13:08:48.937695 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.937670    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 13:08:48.938281 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.938262    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 13:08:48.938368 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.938299    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 13:08:48.942030 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.942008    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 13:08:48.944006 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.943992    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 13:08:48.944074 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.944022    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 13:08:48.946023 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946007    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 13:08:48.946081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946035    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 13:08:48.946081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946044    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 13:08:48.946081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946050    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 13:08:48.946081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946059    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 13:08:48.946081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946065    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 13:08:48.946081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946077    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 13:08:48.946081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946083    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 13:08:48.946352 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946090    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 13:08:48.946352 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946095    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 13:08:48.946352 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946103    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 13:08:48.946352 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946109    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 13:08:48.946352 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946132    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 13:08:48.946352 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.946137    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 13:08:48.948576 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.948559    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-164.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 13:08:48.948655 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:48.948635    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 13:08:48.948702 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:48.948633    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 13:08:48.950106 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.950094    2577 server.go:1268] "Started kubelet"
Sep 16 13:08:48.950222 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.950178    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 13:08:48.950318 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.950266    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 13:08:48.950378 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.950344    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 13:08:48.951067 ip-10-0-134-164 systemd[1]: Started Kubernetes Kubelet.
Sep 16 13:08:48.953597 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.953579    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 13:08:48.954218 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.954205    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 13:08:48.955237 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:48.955213    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 13:08:48.955516 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.955495    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k9726"
Sep 16 13:08:48.957149 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.957132    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 13:08:48.957226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.957157    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 13:08:48.957666 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:48.957643    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:48.957770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.957748    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 13:08:48.957834 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.957789    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 13:08:48.958055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.957931    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 13:08:48.958055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.957972    2577 factory.go:55] Registering systemd factory
Sep 16 13:08:48.958055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.957989    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 16 13:08:48.958055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.958057    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 13:08:48.958268 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.958066    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 13:08:48.958268 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.958213    2577 factory.go:153] Registering CRI-O factory
Sep 16 13:08:48.958268 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.958227    2577 factory.go:223] Registration of the crio container factory successfully
Sep 16 13:08:48.958402 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.958301    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 13:08:48.958402 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.958327    2577 factory.go:103] Registering Raw factory
Sep 16 13:08:48.958402 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.958342    2577 manager.go:1196] Started watching for new ooms in manager
Sep 16 13:08:48.958584 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:48.957590    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-164.ec2.internal.18d5cf1800d98eff  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-164.ec2.internal,UID:ip-10-0-134-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-164.ec2.internal,},FirstTimestamp:2026-09-16 13:08:48.950062847 +0000 UTC m=+0.372804416,LastTimestamp:2026-09-16 13:08:48.950062847 +0000 UTC m=+0.372804416,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-164.ec2.internal,}"
Sep 16 13:08:48.959252 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.959235    2577 manager.go:349] Starting recovery of all containers
Sep 16 13:08:48.961636 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:48.961609    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 13:08:48.961744 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:48.961682    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 13:08:48.963809 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.963787    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k9726"
Sep 16 13:08:48.971567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.971547    2577 manager.go:354] Recovery completed
Sep 16 13:08:48.976471 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.976455    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:08:48.978536 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.978376    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:08:48.978601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.978556    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:08:48.978601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.978573    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:08:48.979149 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.979134    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 13:08:48.979149 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.979147    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 13:08:48.979302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.979158    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:08:48.979302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.979165    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 13:08:48.979940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.979923    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:08:48.980009 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.979953    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:08:48.980009 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.979963    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:08:48.980442 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.980430    2577 policy_none.go:49] "None policy: Start"
Sep 16 13:08:48.980489 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.980447    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 13:08:48.980489 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.980457    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 13:08:48.981299 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:48.981233    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-164.ec2.internal.18d5cf18028c0cf8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-164.ec2.internal,UID:ip-10-0-134-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-164.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-164.ec2.internal,},FirstTimestamp:2026-09-16 13:08:48.97853772 +0000 UTC m=+0.401279293,LastTimestamp:2026-09-16 13:08:48.97853772 +0000 UTC m=+0.401279293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-164.ec2.internal,}"
Sep 16 13:08:48.981587 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:48.981576    2577 policy_none.go:47] "Start"
Sep 16 13:08:49.006472 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.006457    2577 manager.go:337] "Starting Device Plugin manager"
Sep 16 13:08:49.006584 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.006496    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 13:08:49.006584 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.006509    2577 server.go:87] "Starting device plugin registration server"
Sep 16 13:08:49.006869 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.006842    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 13:08:49.006956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.006871    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 13:08:49.007088 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.007065    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 13:08:49.007181 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.007169    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 13:08:49.007261 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.007184    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 13:08:49.007838 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.007749    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 13:08:49.007838 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.007806    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.101365 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.101324    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 13:08:49.102603 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.102581    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 13:08:49.102681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.102648    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 13:08:49.102681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.102679    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 13:08:49.102788 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.102776    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 13:08:49.104804 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.104782    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:08:49.107718 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.107702    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:08:49.108561 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.108546    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:08:49.108638 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.108578    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:08:49.108638 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.108594    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:08:49.108638 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.108618    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.114815 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.114801    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.114881 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.114822    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-164.ec2.internal\": node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.134041 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.133962    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.203454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.203413    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"]
Sep 16 13:08:49.203622 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.203508    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:08:49.204525 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.204506    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:08:49.204631 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.204542    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:08:49.204631 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.204557    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:08:49.206045 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206029    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:08:49.206176 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.206223 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206191    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:08:49.206789 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206773    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:08:49.206789 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206781    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:08:49.206940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206800    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:08:49.206940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206805    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:08:49.206940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206810    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:08:49.206940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.206823    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:08:49.208263 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.208247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.208339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.208275    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:08:49.208918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.208905    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:08:49.209017 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.208931    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:08:49.209017 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.208944    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:08:49.232591 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.232571    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-164.ec2.internal\" not found" node="ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.234399 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.234386    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.237033 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.237016    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-164.ec2.internal\" not found" node="ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.259698 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.259671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c9c34a96dcc4f13ed76996c4a50ec4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"7c9c34a96dcc4f13ed76996c4a50ec4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.259787 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.259707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c9c34a96dcc4f13ed76996c4a50ec4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"7c9c34a96dcc4f13ed76996c4a50ec4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.259787 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.259728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2483e6c65a13c8b4bf2c0714617d8484-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"2483e6c65a13c8b4bf2c0714617d8484\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.335059 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.335025    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.360373 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.360345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c9c34a96dcc4f13ed76996c4a50ec4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"7c9c34a96dcc4f13ed76996c4a50ec4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.360475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.360377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c9c34a96dcc4f13ed76996c4a50ec4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"7c9c34a96dcc4f13ed76996c4a50ec4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.360475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.360414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2483e6c65a13c8b4bf2c0714617d8484-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"2483e6c65a13c8b4bf2c0714617d8484\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.360475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.360443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2483e6c65a13c8b4bf2c0714617d8484-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"2483e6c65a13c8b4bf2c0714617d8484\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.360475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.360439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c9c34a96dcc4f13ed76996c4a50ec4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"7c9c34a96dcc4f13ed76996c4a50ec4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.360610 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.360470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c9c34a96dcc4f13ed76996c4a50ec4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"7c9c34a96dcc4f13ed76996c4a50ec4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.436252 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.436223    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.539783 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.539725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.539930 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.539801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.539930 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.539735    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.643489 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.643450    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.744037 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.743999    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.771132 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.771106    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:08:49.847677 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.847645    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Sep 16 13:08:49.864729 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.864706    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:08:49.901218 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.901180    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 13:08:49.901675 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.901328    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 13:08:49.901675 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.901360    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 13:08:49.901675 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.901379    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 13:08:49.940458 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.940429    2577 apiserver.go:52] "Watching apiserver"
Sep 16 13:08:49.957632 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.957611    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 13:08:49.958032 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.958002    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.965857 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.965837    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 13:08:49.967895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.967873    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-vgrv9","openshift-network-operator/iptables-alerter-wchbl","openshift-cluster-node-tuning-operator/tuned-t7dfq","openshift-dns/node-resolver-hqrcn","openshift-multus/multus-additional-cni-plugins-htcdk","openshift-multus/network-metrics-daemon-dpv4k","openshift-ovn-kubernetes/ovnkube-node-7fssq","kube-system/global-pull-secret-syncer-vf2vf","kube-system/konnectivity-agent-qsn4p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg","openshift-image-registry/node-ca-rm9l6","openshift-multus/multus-rsq78"]
Sep 16 13:08:49.969890 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.969875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:49.970018 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.969994    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:08:49.970306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.970292    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 13:08:49.971896 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.971876    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 13:03:49 +0000 UTC" deadline="2028-04-30 21:49:07.655642255 +0000 UTC"
Sep 16 13:08:49.971956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.971896    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14216h40m17.683748465s"
Sep 16 13:08:49.972006 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.971990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:49.973359 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.973337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:49.974141 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.974123    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 13:08:49.974245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.974230    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Sep 16 13:08:49.974245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.974240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 13:08:49.974328 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.974244    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 13:08:49.974363 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.974346    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:08:49.974920 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.974902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:49.975909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.975894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nnp7v\""
Sep 16 13:08:49.976244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.976227    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rjjxt\""
Sep 16 13:08:49.976315 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.976228    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:08:49.976315 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.976230    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 13:08:49.976418 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.976368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:49.977467 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.977450    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 13:08:49.977546 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.977506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m768p\""
Sep 16 13:08:49.978348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.978322    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 13:08:49.978348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.978340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:49.978512 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.978423    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:08:49.978985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.978968    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 13:08:49.979089 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.978970    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 13:08:49.979293 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.979277    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 13:08:49.979364 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.979302    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 13:08:49.979644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.979630    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2mtf\""
Sep 16 13:08:49.979785 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.979770    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 13:08:49.980380 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.980364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:49.981899 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.981884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:49.981987 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:49.981935    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:08:49.982277 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.982262    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 13:08:49.982511 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.982496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7v5fn\""
Sep 16 13:08:49.982611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.982596    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 13:08:49.982611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.982606    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 13:08:49.982707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.982612    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 13:08:49.982767 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.982750    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 13:08:49.982820 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.982793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 13:08:49.983543 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.983526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:08:49.984611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.984595    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 13:08:49.985316 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.985302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:49.986055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.986040    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 13:08:49.986188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.986172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 13:08:49.986285 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.986270    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s7426\""
Sep 16 13:08:49.987253 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.987238    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 13:08:49.987468 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.987437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:49.987542 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.987510    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 13:08:49.987664 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.987635    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxxbq\""
Sep 16 13:08:49.987777 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.987760    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 13:08:49.989107 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.989091    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"]
Sep 16 13:08:49.989167 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.989127    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"]
Sep 16 13:08:49.989206 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.989182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsq78"
Sep 16 13:08:49.989629 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.989612    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mcv2\""
Sep 16 13:08:49.989730 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.989620    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 13:08:49.989730 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.989627    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 13:08:49.989845 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.989629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 13:08:49.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.991051    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x882g\""
Sep 16 13:08:49.991284 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.991271    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 13:08:49.997021 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:49.996716    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7cqcj"
Sep 16 13:08:50.004988 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.004970    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7cqcj"
Sep 16 13:08:50.058822 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.058803    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 13:08:50.064282 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-socket-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.064397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrx84\" (UniqueName: \"kubernetes.io/projected/5524730b-8e5e-4d46-88a4-245527c7cb70-kube-api-access-zrx84\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.064397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60bd63b3-832a-4c62-a7e9-44fa2a63a540-agent-certs\") pod \"konnectivity-agent-qsn4p\" (UID: \"60bd63b3-832a-4c62-a7e9-44fa2a63a540\") " pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:08:50.064397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-socket-dir-parent\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.064397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-run-netns\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.064397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysctl-conf\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.064633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-cni-multus\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.064633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-daemon-config\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.064633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6ll\" (UniqueName: \"kubernetes.io/projected/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-kube-api-access-vw6ll\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.064633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-cni-netd\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.064633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovnkube-script-lib\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.064633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.064633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5524730b-8e5e-4d46-88a4-245527c7cb70-host-slash\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.064633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5111c756-aa34-4811-a126-504b86fac9d3-dbus\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-cni-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-k8s-cni-cncf-io\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovnkube-config\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-etc-kubernetes\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddv5r\" (UniqueName: \"kubernetes.io/projected/e80fb012-edc0-44a9-9fa1-789d0051f454-kube-api-access-ddv5r\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4195754-df4b-4f6e-90b3-cecdf8415aeb-serviceca\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:50.064985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.064982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-slash\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-systemd\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-ovn\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-log-socket\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-env-overrides\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-cni-bin\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-etc-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysconfig\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-var-lib-kubelet\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-kubelet\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-multus-certs\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f870356-a81b-4032-8134-6a9419c26220-tmp-dir\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-node-log\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76hfc\" (UniqueName: \"kubernetes.io/projected/36e8a420-a282-4900-8b0f-db0c7f264a25-kube-api-access-76hfc\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.065454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-lib-modules\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5524730b-8e5e-4d46-88a4-245527c7cb70-iptables-alerter-script\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e80fb012-edc0-44a9-9fa1-789d0051f454-tmp\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60bd63b3-832a-4c62-a7e9-44fa2a63a540-konnectivity-ca\") pod \"konnectivity-agent-qsn4p\" (UID: \"60bd63b3-832a-4c62-a7e9-44fa2a63a540\") " pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnj2\" (UniqueName: \"kubernetes.io/projected/c4195754-df4b-4f6e-90b3-cecdf8415aeb-kube-api-access-qsnj2\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-hostroot\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-var-lib-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-device-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-run\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-sys\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4195754-df4b-4f6e-90b3-cecdf8415aeb-host\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-systemd-units\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovn-node-metrics-cert\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ppb\" (UniqueName: \"kubernetes.io/projected/774031d1-7359-4fd8-aa1a-23ef54aaf426-kube-api-access-l5ppb\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-tuned\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.065990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-system-cni-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.066136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-netns\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-modprobe-d\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysctl-d\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-systemd\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5111c756-aa34-4811-a126-504b86fac9d3-kubelet-config\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-cnibin\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-cni-binary-copy\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-conf-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-host\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-kubernetes\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkf9s\" (UniqueName: \"kubernetes.io/projected/3f870356-a81b-4032-8134-6a9419c26220-kube-api-access-qkf9s\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-system-cni-dir\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6wh\" (UniqueName: \"kubernetes.io/projected/19312944-1bba-4df0-b850-4849fbb74797-kube-api-access-5l6wh\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5724\" (UniqueName: \"kubernetes.io/projected/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-kube-api-access-z5724\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.066832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cnibin\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.067423 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-kubelet\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.067423 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-cni-bin\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.067423 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-registration-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.067423 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-sys-fs\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.067423 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.067423 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.066678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f870356-a81b-4032-8134-6a9419c26220-hosts-file\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.167508 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.167508 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-slash\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-systemd\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-ovn\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-log-socket\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-slash\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-env-overrides\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.167655    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-ovn\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-cni-bin\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-log-socket\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.167741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-systemd\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-cni-bin\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-etc-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.167848    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs podName:19312944-1bba-4df0-b850-4849fbb74797 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:50.667830631 +0000 UTC m=+2.090572211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs") pod "network-metrics-daemon-dpv4k" (UID: "19312944-1bba-4df0-b850-4849fbb74797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-etc-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysconfig\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.167982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-var-lib-kubelet\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-kubelet\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysconfig\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-multus-certs\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-kubelet\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f870356-a81b-4032-8134-6a9419c26220-tmp-dir\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-var-lib-kubelet\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-multus-certs\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.168243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-env-overrides\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-node-log\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76hfc\" (UniqueName: \"kubernetes.io/projected/36e8a420-a282-4900-8b0f-db0c7f264a25-kube-api-access-76hfc\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-lib-modules\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5524730b-8e5e-4d46-88a4-245527c7cb70-iptables-alerter-script\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e80fb012-edc0-44a9-9fa1-789d0051f454-tmp\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-node-log\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60bd63b3-832a-4c62-a7e9-44fa2a63a540-konnectivity-ca\") pod \"konnectivity-agent-qsn4p\" (UID: \"60bd63b3-832a-4c62-a7e9-44fa2a63a540\") " pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f870356-a81b-4032-8134-6a9419c26220-tmp-dir\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnj2\" (UniqueName: \"kubernetes.io/projected/c4195754-df4b-4f6e-90b3-cecdf8415aeb-kube-api-access-qsnj2\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-hostroot\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-lib-modules\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-var-lib-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-device-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-hostroot\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-run\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-sys\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-var-lib-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4195754-df4b-4f6e-90b3-cecdf8415aeb-host\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-systemd-units\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-run\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovn-node-metrics-cert\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-device-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ppb\" (UniqueName: \"kubernetes.io/projected/774031d1-7359-4fd8-aa1a-23ef54aaf426-kube-api-access-l5ppb\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-systemd-units\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-sys\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-tuned\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-system-cni-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-netns\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168826    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5524730b-8e5e-4d46-88a4-245527c7cb70-iptables-alerter-script\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.169895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4195754-df4b-4f6e-90b3-cecdf8415aeb-host\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-modprobe-d\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysctl-d\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-system-cni-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.168992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-systemd\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-netns\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5111c756-aa34-4811-a126-504b86fac9d3-kubelet-config\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-systemd\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-cnibin\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5111c756-aa34-4811-a126-504b86fac9d3-kubelet-config\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysctl-d\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-cni-binary-copy\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-modprobe-d\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-conf-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-cnibin\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60bd63b3-832a-4c62-a7e9-44fa2a63a540-konnectivity-ca\") pod \"konnectivity-agent-qsn4p\" (UID: \"60bd63b3-832a-4c62-a7e9-44fa2a63a540\") " pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-conf-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.170770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-host\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-kubernetes\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkf9s\" (UniqueName: \"kubernetes.io/projected/3f870356-a81b-4032-8134-6a9419c26220-kube-api-access-qkf9s\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.169952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-host\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-system-cni-dir\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6wh\" (UniqueName: \"kubernetes.io/projected/19312944-1bba-4df0-b850-4849fbb74797-kube-api-access-5l6wh\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5724\" (UniqueName: \"kubernetes.io/projected/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-kube-api-access-z5724\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-kubernetes\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-cni-binary-copy\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cnibin\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-kubelet\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-cni-bin\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-registration-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-kubelet\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-sys-fs\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.171677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f870356-a81b-4032-8134-6a9419c26220-hosts-file\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-socket-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrx84\" (UniqueName: \"kubernetes.io/projected/5524730b-8e5e-4d46-88a4-245527c7cb70-kube-api-access-zrx84\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60bd63b3-832a-4c62-a7e9-44fa2a63a540-agent-certs\") pod \"konnectivity-agent-qsn4p\" (UID: \"60bd63b3-832a-4c62-a7e9-44fa2a63a540\") " pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-socket-dir-parent\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-run-netns\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.170656    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-registration-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-cni-bin\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysctl-conf\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.170979    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret podName:5111c756-aa34-4811-a126-504b86fac9d3 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:50.670954322 +0000 UTC m=+2.093695901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret") pod "global-pull-secret-syncer-vf2vf" (UID: "5111c756-aa34-4811-a126-504b86fac9d3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-sys-fs\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-socket-dir\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-sysctl-conf\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-cni-multus\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-daemon-config\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.172554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-run-netns\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6ll\" (UniqueName: \"kubernetes.io/projected/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-kube-api-access-vw6ll\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-cni-netd\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-host-cni-netd\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.171995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-system-cni-dir\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-var-lib-cni-multus\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.170322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cnibin\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-daemon-config\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f870356-a81b-4032-8134-6a9419c26220-hosts-file\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovnkube-script-lib\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5524730b-8e5e-4d46-88a4-245527c7cb70-host-slash\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5111c756-aa34-4811-a126-504b86fac9d3-dbus\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-cni-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-k8s-cni-cncf-io\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-socket-dir-parent\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.173490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovnkube-config\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-etc-kubernetes\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddv5r\" (UniqueName: \"kubernetes.io/projected/e80fb012-edc0-44a9-9fa1-789d0051f454-kube-api-access-ddv5r\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4195754-df4b-4f6e-90b3-cecdf8415aeb-serviceca\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovnkube-script-lib\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.172865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-multus-cni-dir\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5111c756-aa34-4811-a126-504b86fac9d3-dbus\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5524730b-8e5e-4d46-88a4-245527c7cb70-host-slash\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-host-run-k8s-cni-cncf-io\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4195754-df4b-4f6e-90b3-cecdf8415aeb-serviceca\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36e8a420-a282-4900-8b0f-db0c7f264a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-etc-kubernetes\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/774031d1-7359-4fd8-aa1a-23ef54aaf426-run-openvswitch\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.173754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovnkube-config\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.174337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.174059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e80fb012-edc0-44a9-9fa1-789d0051f454-tmp\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.175751 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.175348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e80fb012-edc0-44a9-9fa1-789d0051f454-etc-tuned\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.175856 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.175822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/774031d1-7359-4fd8-aa1a-23ef54aaf426-ovn-node-metrics-cert\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.176182 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.176141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.177203 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.177181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60bd63b3-832a-4c62-a7e9-44fa2a63a540-agent-certs\") pod \"konnectivity-agent-qsn4p\" (UID: \"60bd63b3-832a-4c62-a7e9-44fa2a63a540\") " pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:08:50.179460 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.179437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnj2\" (UniqueName: \"kubernetes.io/projected/c4195754-df4b-4f6e-90b3-cecdf8415aeb-kube-api-access-qsnj2\") pod \"node-ca-rm9l6\" (UID: \"c4195754-df4b-4f6e-90b3-cecdf8415aeb\") " pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.179641 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.179624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76hfc\" (UniqueName: \"kubernetes.io/projected/36e8a420-a282-4900-8b0f-db0c7f264a25-kube-api-access-76hfc\") pod \"aws-ebs-csi-driver-node-2d6mg\" (UID: \"36e8a420-a282-4900-8b0f-db0c7f264a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.188324 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.188305    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:08:50.188394 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.188328    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:08:50.188394 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.188339    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ngfg5 for pod openshift-network-diagnostics/network-check-target-vgrv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:50.188498 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.188446    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5 podName:86f38672-4e25-4e3e-bc7c-e2aee1294de1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:50.688431334 +0000 UTC m=+2.111172892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ngfg5" (UniqueName: "kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5") pod "network-check-target-vgrv9" (UID: "86f38672-4e25-4e3e-bc7c-e2aee1294de1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:50.190540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.190518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5724\" (UniqueName: \"kubernetes.io/projected/3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2-kube-api-access-z5724\") pod \"multus-rsq78\" (UID: \"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2\") " pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.190666 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.190646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkf9s\" (UniqueName: \"kubernetes.io/projected/3f870356-a81b-4032-8134-6a9419c26220-kube-api-access-qkf9s\") pod \"node-resolver-hqrcn\" (UID: \"3f870356-a81b-4032-8134-6a9419c26220\") " pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.193415 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.193391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddv5r\" (UniqueName: \"kubernetes.io/projected/e80fb012-edc0-44a9-9fa1-789d0051f454-kube-api-access-ddv5r\") pod \"tuned-t7dfq\" (UID: \"e80fb012-edc0-44a9-9fa1-789d0051f454\") " pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.193881 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.193855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ppb\" (UniqueName: \"kubernetes.io/projected/774031d1-7359-4fd8-aa1a-23ef54aaf426-kube-api-access-l5ppb\") pod \"ovnkube-node-7fssq\" (UID: \"774031d1-7359-4fd8-aa1a-23ef54aaf426\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.194224 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.194203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6ll\" (UniqueName: \"kubernetes.io/projected/a7e4d7bb-a82f-4811-8a44-0c3ac78869b7-kube-api-access-vw6ll\") pod \"multus-additional-cni-plugins-htcdk\" (UID: \"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7\") " pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.194310 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.194285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrx84\" (UniqueName: \"kubernetes.io/projected/5524730b-8e5e-4d46-88a4-245527c7cb70-kube-api-access-zrx84\") pod \"iptables-alerter-wchbl\" (UID: \"5524730b-8e5e-4d46-88a4-245527c7cb70\") " pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.194366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.194327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6wh\" (UniqueName: \"kubernetes.io/projected/19312944-1bba-4df0-b850-4849fbb74797-kube-api-access-5l6wh\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:50.303724 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.303654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wchbl"
Sep 16 13:08:50.321112 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.321087    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:08:50.326436 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.326419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t7dfq"
Sep 16 13:08:50.347408 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.347379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hqrcn"
Sep 16 13:08:50.364495 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.364469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-htcdk"
Sep 16 13:08:50.372848 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.372831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:08:50.375842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.375826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:08:50.399784 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.399764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg"
Sep 16 13:08:50.413300 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.413249    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e8a420_a282_4900_8b0f_db0c7f264a25.slice/crio-545e72ab42d9639e91297526d538db90fbba5d54879a2bfa8d637c122c8221c0 WatchSource:0}: Error finding container 545e72ab42d9639e91297526d538db90fbba5d54879a2bfa8d637c122c8221c0: Status 404 returned error can't find the container with id 545e72ab42d9639e91297526d538db90fbba5d54879a2bfa8d637c122c8221c0
Sep 16 13:08:50.413300 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.413285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rm9l6"
Sep 16 13:08:50.413620 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.413593    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2483e6c65a13c8b4bf2c0714617d8484.slice/crio-911465f6b3dc91842c1670bf47677e404c562db006028aafda57b85525f6dea0 WatchSource:0}: Error finding container 911465f6b3dc91842c1670bf47677e404c562db006028aafda57b85525f6dea0: Status 404 returned error can't find the container with id 911465f6b3dc91842c1670bf47677e404c562db006028aafda57b85525f6dea0
Sep 16 13:08:50.413935 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.413911    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f870356_a81b_4032_8134_6a9419c26220.slice/crio-e11c54ac093577665f6687ea251ebd42342b8bfe3bf2ef8bcf8b100ff653f15a WatchSource:0}: Error finding container e11c54ac093577665f6687ea251ebd42342b8bfe3bf2ef8bcf8b100ff653f15a: Status 404 returned error can't find the container with id e11c54ac093577665f6687ea251ebd42342b8bfe3bf2ef8bcf8b100ff653f15a
Sep 16 13:08:50.414524 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.414487    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9c34a96dcc4f13ed76996c4a50ec4c.slice/crio-d1880e9be0a0cf67b88aa413f56c49b6f1805e5a2beb327a041c969a0c7e11da WatchSource:0}: Error finding container d1880e9be0a0cf67b88aa413f56c49b6f1805e5a2beb327a041c969a0c7e11da: Status 404 returned error can't find the container with id d1880e9be0a0cf67b88aa413f56c49b6f1805e5a2beb327a041c969a0c7e11da
Sep 16 13:08:50.415239 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.415206    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bd63b3_832a_4c62_a7e9_44fa2a63a540.slice/crio-6f6f920828bc486344e39ddbb9eafa744f1389ba80b6399f6a6ae72f38baac56 WatchSource:0}: Error finding container 6f6f920828bc486344e39ddbb9eafa744f1389ba80b6399f6a6ae72f38baac56: Status 404 returned error can't find the container with id 6f6f920828bc486344e39ddbb9eafa744f1389ba80b6399f6a6ae72f38baac56
Sep 16 13:08:50.416335 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.416321    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e4d7bb_a82f_4811_8a44_0c3ac78869b7.slice/crio-7fa69f64b4f1884724423c953dbb6a1bdd7b1fbe92d5b09b60a97adf0021226d WatchSource:0}: Error finding container 7fa69f64b4f1884724423c953dbb6a1bdd7b1fbe92d5b09b60a97adf0021226d: Status 404 returned error can't find the container with id 7fa69f64b4f1884724423c953dbb6a1bdd7b1fbe92d5b09b60a97adf0021226d
Sep 16 13:08:50.417474 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.417447    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774031d1_7359_4fd8_aa1a_23ef54aaf426.slice/crio-d84683676f3ef8ac5882ed12fa1a95724c804cbc55540892a4b108cffddedd6c WatchSource:0}: Error finding container d84683676f3ef8ac5882ed12fa1a95724c804cbc55540892a4b108cffddedd6c: Status 404 returned error can't find the container with id d84683676f3ef8ac5882ed12fa1a95724c804cbc55540892a4b108cffddedd6c
Sep 16 13:08:50.419635 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.419611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsq78"
Sep 16 13:08:50.422301 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.422169    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:08:50.426767 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.426731    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4195754_df4b_4f6e_90b3_cecdf8415aeb.slice/crio-4aee495989051f3ef783bf6aa052c3f050067a76ec7feaa55c95f03719ac3ecd WatchSource:0}: Error finding container 4aee495989051f3ef783bf6aa052c3f050067a76ec7feaa55c95f03719ac3ecd: Status 404 returned error can't find the container with id 4aee495989051f3ef783bf6aa052c3f050067a76ec7feaa55c95f03719ac3ecd
Sep 16 13:08:50.428848 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.428820    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1b934e_e6c0_42f7_8910_5ed9ee47c0c2.slice/crio-a8cd39c4420dc8c6d5094b40546208c82bc034d28f967d65fe9beecaf5a71637 WatchSource:0}: Error finding container a8cd39c4420dc8c6d5094b40546208c82bc034d28f967d65fe9beecaf5a71637: Status 404 returned error can't find the container with id a8cd39c4420dc8c6d5094b40546208c82bc034d28f967d65fe9beecaf5a71637
Sep 16 13:08:50.648926 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.648895    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80fb012_edc0_44a9_9fa1_789d0051f454.slice/crio-40914c617fb86ca06c71ab79cd516bd50d3bcea6449202e175c7933608a5a5ac WatchSource:0}: Error finding container 40914c617fb86ca06c71ab79cd516bd50d3bcea6449202e175c7933608a5a5ac: Status 404 returned error can't find the container with id 40914c617fb86ca06c71ab79cd516bd50d3bcea6449202e175c7933608a5a5ac
Sep 16 13:08:50.677080 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.677048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:50.677206 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.677110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:50.677206 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.677182    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:50.677206 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.677196    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:50.677348 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.677244    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret podName:5111c756-aa34-4811-a126-504b86fac9d3 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:51.677230365 +0000 UTC m=+3.099971929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret") pod "global-pull-secret-syncer-vf2vf" (UID: "5111c756-aa34-4811-a126-504b86fac9d3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:50.677348 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.677259    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs podName:19312944-1bba-4df0-b850-4849fbb74797 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:51.677252201 +0000 UTC m=+3.099993759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs") pod "network-metrics-daemon-dpv4k" (UID: "19312944-1bba-4df0-b850-4849fbb74797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:50.778449 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:50.778413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:50.778601 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.778586    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:08:50.778666 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.778610    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:08:50.778666 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.778620    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ngfg5 for pod openshift-network-diagnostics/network-check-target-vgrv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:50.778733 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:50.778679    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5 podName:86f38672-4e25-4e3e-bc7c-e2aee1294de1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:51.778659204 +0000 UTC m=+3.201400780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngfg5" (UniqueName: "kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5") pod "network-check-target-vgrv9" (UID: "86f38672-4e25-4e3e-bc7c-e2aee1294de1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:50.827383 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:08:50.827354    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5524730b_8e5e_4d46_88a4_245527c7cb70.slice/crio-7818729d943cdbbc063994850849aa2c109e944794147cc18071b2fcb67e28fe WatchSource:0}: Error finding container 7818729d943cdbbc063994850849aa2c109e944794147cc18071b2fcb67e28fe: Status 404 returned error can't find the container with id 7818729d943cdbbc063994850849aa2c109e944794147cc18071b2fcb67e28fe
Sep 16 13:08:51.007104 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.007020    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:03:50 +0000 UTC" deadline="2028-06-14 08:22:17.589639366 +0000 UTC"
Sep 16 13:08:51.007104 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.007074    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15283h13m26.58257081s"
Sep 16 13:08:51.106640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.105942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:51.106640 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.106075    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:08:51.112894 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.112810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"7c9c34a96dcc4f13ed76996c4a50ec4c","Type":"ContainerStarted","Data":"d1880e9be0a0cf67b88aa413f56c49b6f1805e5a2beb327a041c969a0c7e11da"}
Sep 16 13:08:51.115442 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.115410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rm9l6" event={"ID":"c4195754-df4b-4f6e-90b3-cecdf8415aeb","Type":"ContainerStarted","Data":"4aee495989051f3ef783bf6aa052c3f050067a76ec7feaa55c95f03719ac3ecd"}
Sep 16 13:08:51.117265 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.117239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg" event={"ID":"36e8a420-a282-4900-8b0f-db0c7f264a25","Type":"ContainerStarted","Data":"545e72ab42d9639e91297526d538db90fbba5d54879a2bfa8d637c122c8221c0"}
Sep 16 13:08:51.119131 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.119103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qsn4p" event={"ID":"60bd63b3-832a-4c62-a7e9-44fa2a63a540","Type":"ContainerStarted","Data":"6f6f920828bc486344e39ddbb9eafa744f1389ba80b6399f6a6ae72f38baac56"}
Sep 16 13:08:51.121890 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.121855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerStarted","Data":"7fa69f64b4f1884724423c953dbb6a1bdd7b1fbe92d5b09b60a97adf0021226d"}
Sep 16 13:08:51.125174 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.125147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hqrcn" event={"ID":"3f870356-a81b-4032-8134-6a9419c26220","Type":"ContainerStarted","Data":"e11c54ac093577665f6687ea251ebd42342b8bfe3bf2ef8bcf8b100ff653f15a"}
Sep 16 13:08:51.127787 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.127761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wchbl" event={"ID":"5524730b-8e5e-4d46-88a4-245527c7cb70","Type":"ContainerStarted","Data":"7818729d943cdbbc063994850849aa2c109e944794147cc18071b2fcb67e28fe"}
Sep 16 13:08:51.132371 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.132339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t7dfq" event={"ID":"e80fb012-edc0-44a9-9fa1-789d0051f454","Type":"ContainerStarted","Data":"40914c617fb86ca06c71ab79cd516bd50d3bcea6449202e175c7933608a5a5ac"}
Sep 16 13:08:51.138341 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.138313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsq78" event={"ID":"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2","Type":"ContainerStarted","Data":"a8cd39c4420dc8c6d5094b40546208c82bc034d28f967d65fe9beecaf5a71637"}
Sep 16 13:08:51.144310 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.144280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"d84683676f3ef8ac5882ed12fa1a95724c804cbc55540892a4b108cffddedd6c"}
Sep 16 13:08:51.147529 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.147500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal" event={"ID":"2483e6c65a13c8b4bf2c0714617d8484","Type":"ContainerStarted","Data":"911465f6b3dc91842c1670bf47677e404c562db006028aafda57b85525f6dea0"}
Sep 16 13:08:51.190500 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.190467    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:08:51.686247 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.686206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:51.686463 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.686313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:51.686528 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.686490    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:51.686584 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.686563    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret podName:5111c756-aa34-4811-a126-504b86fac9d3 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:53.686541842 +0000 UTC m=+5.109283421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret") pod "global-pull-secret-syncer-vf2vf" (UID: "5111c756-aa34-4811-a126-504b86fac9d3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:51.687133 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.687107    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:51.687235 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.687175    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs podName:19312944-1bba-4df0-b850-4849fbb74797 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:53.687156615 +0000 UTC m=+5.109898180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs") pod "network-metrics-daemon-dpv4k" (UID: "19312944-1bba-4df0-b850-4849fbb74797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:51.755918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.755614    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:08:51.787441 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:51.787403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:51.787638 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.787598    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:08:51.787638 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.787620    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:08:51.787761 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.787643    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ngfg5 for pod openshift-network-diagnostics/network-check-target-vgrv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:51.787761 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:51.787705    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5 podName:86f38672-4e25-4e3e-bc7c-e2aee1294de1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:53.78768441 +0000 UTC m=+5.210425987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngfg5" (UniqueName: "kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5") pod "network-check-target-vgrv9" (UID: "86f38672-4e25-4e3e-bc7c-e2aee1294de1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:52.007974 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:52.007919    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:03:50 +0000 UTC" deadline="2028-04-29 17:52:34.680258836 +0000 UTC"
Sep 16 13:08:52.007974 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:52.007964    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14188h43m42.672299926s"
Sep 16 13:08:52.104660 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:52.103904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:52.104660 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:52.104051    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:08:52.104660 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:52.104495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:52.104660 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:52.104600    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:08:52.468869 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:52.468794    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:08:53.106041 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:53.106003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:53.106553 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.106137    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:08:53.708658 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:53.707604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:53.708658 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:53.707921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:53.708658 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.708112    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:53.708658 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.708178    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs podName:19312944-1bba-4df0-b850-4849fbb74797 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:57.708156441 +0000 UTC m=+9.130898002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs") pod "network-metrics-daemon-dpv4k" (UID: "19312944-1bba-4df0-b850-4849fbb74797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:53.708658 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.708567    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:53.708658 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.708613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret podName:5111c756-aa34-4811-a126-504b86fac9d3 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:57.708597744 +0000 UTC m=+9.131339309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret") pod "global-pull-secret-syncer-vf2vf" (UID: "5111c756-aa34-4811-a126-504b86fac9d3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:53.809272 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:53.808484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:53.809272 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.808690    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:08:53.809272 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.808713    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:08:53.809272 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.808728    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ngfg5 for pod openshift-network-diagnostics/network-check-target-vgrv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:53.809272 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:53.808786    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5 podName:86f38672-4e25-4e3e-bc7c-e2aee1294de1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:57.808767692 +0000 UTC m=+9.231509256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngfg5" (UniqueName: "kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5") pod "network-check-target-vgrv9" (UID: "86f38672-4e25-4e3e-bc7c-e2aee1294de1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:54.104286 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:54.103485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:54.104286 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:54.103630    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:08:54.104286 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:54.104060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:54.104286 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:54.104179    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:08:55.104073 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:55.103509    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:55.104073 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:55.103659    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:08:56.103361 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:56.103288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:56.103568 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:56.103434    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:08:56.103568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:56.103493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:56.103696 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:56.103613    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:08:57.103080 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:57.103036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:57.103577 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.103193    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:08:57.745983 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:57.745933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:57.746159 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.746112    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:57.746159 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:57.746138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:57.746299 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.746188    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret podName:5111c756-aa34-4811-a126-504b86fac9d3 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:05.746164795 +0000 UTC m=+17.168906376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret") pod "global-pull-secret-syncer-vf2vf" (UID: "5111c756-aa34-4811-a126-504b86fac9d3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:08:57.746299 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.746255    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:57.746414 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.746322    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs podName:19312944-1bba-4df0-b850-4849fbb74797 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:05.746304473 +0000 UTC m=+17.169046035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs") pod "network-metrics-daemon-dpv4k" (UID: "19312944-1bba-4df0-b850-4849fbb74797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:08:57.847417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:57.847324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:57.847580 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.847525    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:08:57.847580 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.847553    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:08:57.847580 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.847568    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ngfg5 for pod openshift-network-diagnostics/network-check-target-vgrv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:57.847750 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:57.847635    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5 podName:86f38672-4e25-4e3e-bc7c-e2aee1294de1 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:05.847616793 +0000 UTC m=+17.270358374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngfg5" (UniqueName: "kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5") pod "network-check-target-vgrv9" (UID: "86f38672-4e25-4e3e-bc7c-e2aee1294de1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:08:58.103568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:58.102919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:08:58.103568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:58.102937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:08:58.103568 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:58.103066    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:08:58.103568 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:58.103159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:08:59.106975 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:08:59.106943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:08:59.107544 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:08:59.107066    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:00.103557 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:00.103521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:00.103770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:00.103576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:00.103770 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:00.103680    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:00.104103 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:00.104054    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:01.103296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:01.103260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:01.103732 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:01.103410    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:02.103466 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:02.103424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:02.103924 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:02.103479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:02.103924 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:02.103552    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:02.103924 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:02.103664    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:03.103822 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:03.103800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:03.104191 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:03.103936    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:04.103830 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:04.103790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:04.104283 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:04.103846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:04.104283 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:04.103960    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:04.104283 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:04.104085    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:05.103551 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:05.103514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:05.103705 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.103645    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:05.812609 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:05.812576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:05.813023 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:05.812653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:05.813023 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.812744    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:09:05.813023 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.812745    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:09:05.813023 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.812801    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret podName:5111c756-aa34-4811-a126-504b86fac9d3 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:21.812782498 +0000 UTC m=+33.235524055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret") pod "global-pull-secret-syncer-vf2vf" (UID: "5111c756-aa34-4811-a126-504b86fac9d3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:09:05.813023 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.812816    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs podName:19312944-1bba-4df0-b850-4849fbb74797 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:21.812808849 +0000 UTC m=+33.235550407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs") pod "network-metrics-daemon-dpv4k" (UID: "19312944-1bba-4df0-b850-4849fbb74797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:09:05.913643 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:05.913611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:05.913833 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.913804    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:09:05.913833 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.913831    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:09:05.913973 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.913846    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ngfg5 for pod openshift-network-diagnostics/network-check-target-vgrv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:09:05.913973 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:05.913926    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5 podName:86f38672-4e25-4e3e-bc7c-e2aee1294de1 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:21.913905794 +0000 UTC m=+33.336647368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngfg5" (UniqueName: "kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5") pod "network-check-target-vgrv9" (UID: "86f38672-4e25-4e3e-bc7c-e2aee1294de1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:09:06.103475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:06.103347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:06.103640 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:06.103508    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:06.103640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:06.103552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:06.103746 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:06.103690    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:07.103489 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:07.103451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:07.103990 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:07.103582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:08.103369 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:08.103340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:08.103544 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:08.103340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:08.103544 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:08.103470    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:08.103544 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:08.103528    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:09.104354 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:09.104314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:09.104828 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:09.104420    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:10.103115 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:10.103076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:10.103302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:10.103124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:10.103302 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:10.103220    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:10.103410 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:10.103354    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:11.105362 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.104683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:11.105362 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:11.104991    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:11.200489 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.200388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal" event={"ID":"2483e6c65a13c8b4bf2c0714617d8484","Type":"ContainerStarted","Data":"8fbe1650c51242b92a5704151b6ec070b2eff8f5f17558b038babc0517fc9d48"}
Sep 16 13:09:11.202351 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.202308    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c9c34a96dcc4f13ed76996c4a50ec4c" containerID="b35dd7a9f65d41ed3a898545d23a6b4957e145d38ba591a99c3daa56a9a054c6" exitCode=0
Sep 16 13:09:11.202459 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.202408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"7c9c34a96dcc4f13ed76996c4a50ec4c","Type":"ContainerDied","Data":"b35dd7a9f65d41ed3a898545d23a6b4957e145d38ba591a99c3daa56a9a054c6"}
Sep 16 13:09:11.204139 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.204113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rm9l6" event={"ID":"c4195754-df4b-4f6e-90b3-cecdf8415aeb","Type":"ContainerStarted","Data":"3ec3cc2f7d4c0baee642f8a9a63414c6a44052ba20aecb78b4408d839e0c1132"}
Sep 16 13:09:11.205771 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.205752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg" event={"ID":"36e8a420-a282-4900-8b0f-db0c7f264a25","Type":"ContainerStarted","Data":"d5dfb85581b551845128fc87ebb7f80be50fe89ef466ca2224b050fbf5ed0d2c"}
Sep 16 13:09:11.207446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.207411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qsn4p" event={"ID":"60bd63b3-832a-4c62-a7e9-44fa2a63a540","Type":"ContainerStarted","Data":"2089c6b7e582353dd63ea1a9e53c8d217d608fee4cb5fca7cade9c0c77768835"}
Sep 16 13:09:11.209159 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.209132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerStarted","Data":"96acee2018f34d44c9f0d83ef60d23bd3132ded13c8641c425567f8db34abaa5"}
Sep 16 13:09:11.210707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.210684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hqrcn" event={"ID":"3f870356-a81b-4032-8134-6a9419c26220","Type":"ContainerStarted","Data":"e1dc4f7307cefaff62643099a29824acfa2c07dca4827aba6d9f166ff61b66ba"}
Sep 16 13:09:11.212273 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.212253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t7dfq" event={"ID":"e80fb012-edc0-44a9-9fa1-789d0051f454","Type":"ContainerStarted","Data":"8e251ca30defcb22326f7673aad92d175df49310f3082ac1ce45e2a48b594b4d"}
Sep 16 13:09:11.213830 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.213788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsq78" event={"ID":"3f1b934e-e6c0-42f7-8910-5ed9ee47c0c2","Type":"ContainerStarted","Data":"82621286f08161adf8056109d58815957001b4e8859d55dd2c1648002317d7bc"}
Sep 16 13:09:11.215930 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.215906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"bc7f65385e576faefea6492bafdb4a2e3f329b2b30bc5fff1e38e94c636794f2"}
Sep 16 13:09:11.216029 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.215983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"51babf75ac2110b100ea66b6988c6e4257848d1437605582824535e7c189ba0e"}
Sep 16 13:09:11.216084 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.216054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"c9d77900699fab111b26ef41e27afa37f16c0894a869ffe9b39876c48c1e160f"}
Sep 16 13:09:11.236632 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.236592    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rsq78" podStartSLOduration=1.000142215 podStartE2EDuration="21.236580031s" totalImagesPullingTime="20.236437816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:11.236470244 +0000 UTC m=+22.659211823" watchObservedRunningTime="2026-09-16 13:09:11.236580031 +0000 UTC m=+22.659321610"
Sep 16 13:09:11.237081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.237058    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal" podStartSLOduration=21.237051192 podStartE2EDuration="21.237051192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:11.219644919 +0000 UTC m=+22.642386500" watchObservedRunningTime="2026-09-16 13:09:11.237051192 +0000 UTC m=+22.659792772"
Sep 16 13:09:11.261245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.261208    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t7dfq" podStartSLOduration=1.27960839 podStartE2EDuration="21.261193575s" totalImagesPullingTime="19.981585185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:11.260997263 +0000 UTC m=+22.683738843" watchObservedRunningTime="2026-09-16 13:09:11.261193575 +0000 UTC m=+22.683935154"
Sep 16 13:09:11.295554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.295503    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hqrcn" podStartSLOduration=1.087755162 podStartE2EDuration="21.295483511s" totalImagesPullingTime="20.207728349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:11.294964091 +0000 UTC m=+22.717705670" watchObservedRunningTime="2026-09-16 13:09:11.295483511 +0000 UTC m=+22.718225091"
Sep 16 13:09:11.371673 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.371628    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qsn4p" podStartSLOduration=1.163038362 podStartE2EDuration="21.371612919s" totalImagesPullingTime="20.208574557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:11.371171156 +0000 UTC m=+22.793912736" watchObservedRunningTime="2026-09-16 13:09:11.371612919 +0000 UTC m=+22.794354497"
Sep 16 13:09:11.390208 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.390164    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rm9l6" podStartSLOduration=1.188746645 podStartE2EDuration="21.390150161s" totalImagesPullingTime="20.201403516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:11.389801114 +0000 UTC m=+22.812542693" watchObservedRunningTime="2026-09-16 13:09:11.390150161 +0000 UTC m=+22.812891747"
Sep 16 13:09:11.646046 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.646011    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:09:11.646652 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:11.646630    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:09:12.103112 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.103082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:12.103272 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.103090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:12.103272 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:12.103225    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:12.103387 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:12.103285    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:12.221135 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.221104    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e4d7bb-a82f-4811-8a44-0c3ac78869b7" containerID="96acee2018f34d44c9f0d83ef60d23bd3132ded13c8641c425567f8db34abaa5" exitCode=0
Sep 16 13:09:12.221686 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.221187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerDied","Data":"96acee2018f34d44c9f0d83ef60d23bd3132ded13c8641c425567f8db34abaa5"}
Sep 16 13:09:12.223964 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.223524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wchbl" event={"ID":"5524730b-8e5e-4d46-88a4-245527c7cb70","Type":"ContainerStarted","Data":"4a753dcc603a24d5e8c67a05bf99c08b0f227437cdc40ab56b306575e225adb5"}
Sep 16 13:09:12.227311 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.227284    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"35667c176996c0d7496df2d0460c97353f043d6e7ebb405b341fc20e5e6d6716"}
Sep 16 13:09:12.227416 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.227366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"24c04a9860490d9304aace76a62538107e466c7c0daccfe67e000725fefe7c3e"}
Sep 16 13:09:12.227467 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.227415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"901c3b95351c02e3a4f310acbbc5b37b9434b03db8059967d94ec440b22cccc6"}
Sep 16 13:09:12.229734 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.229708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"7c9c34a96dcc4f13ed76996c4a50ec4c","Type":"ContainerStarted","Data":"3dee2134e4e02850a1d0169023ed7f1cb7c330cd4bd78dbdedb436950cb860b9"}
Sep 16 13:09:12.230422 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.230403    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:09:12.231387 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.231364    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qsn4p"
Sep 16 13:09:12.239933 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.239884    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wchbl" podStartSLOduration=2.437616332 podStartE2EDuration="22.239832285s" totalImagesPullingTime="19.802215953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:12.239159648 +0000 UTC m=+23.661901227" watchObservedRunningTime="2026-09-16 13:09:12.239832285 +0000 UTC m=+23.662573867"
Sep 16 13:09:12.268837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:12.268790    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" podStartSLOduration=22.268774616 podStartE2EDuration="22.268774616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:12.268616301 +0000 UTC m=+23.691357879" watchObservedRunningTime="2026-09-16 13:09:12.268774616 +0000 UTC m=+23.691516191"
Sep 16 13:09:13.103864 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:13.103819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:13.104055 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:13.103966    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:13.228616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:13.228591    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 13:09:13.232881 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:13.232838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg" event={"ID":"36e8a420-a282-4900-8b0f-db0c7f264a25","Type":"ContainerStarted","Data":"415decb3a8ad97b3804ce7103214da034930f98d9651eaf5dc8dff92f6ce404c"}
Sep 16 13:09:14.019703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:14.019570    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T13:09:13.228610536Z","UUID":"e2c2733a-60a8-4c75-b432-0bf06af08dad","Handler":null,"Name":"","Endpoint":""}
Sep 16 13:09:14.021365 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:14.021339    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 13:09:14.021496 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:14.021383    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 13:09:14.102947 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:14.102915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:14.103116 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:14.102928    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:14.103116 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:14.103024    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:14.103116 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:14.103094    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:14.239961 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:14.239923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"36b57b8359926e515533bd8144c66a5c2dd75fc7cf7edbd937ca83d079005a16"}
Sep 16 13:09:15.103095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:15.103065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:15.103399 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:15.103194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:16.103049 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:16.102970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:16.103049 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:16.103016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:16.103517 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:16.103120    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:16.103517 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:16.103284    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:17.103758 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.103615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:17.104043 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:17.103832    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:17.253007 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.252974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" event={"ID":"774031d1-7359-4fd8-aa1a-23ef54aaf426","Type":"ContainerStarted","Data":"b63f6b53ba78f014726961945a5d4091f3843e45f39e92b775abcaa3886de7fb"}
Sep 16 13:09:17.253160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.253120    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:09:17.253216 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.253190    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:09:17.253269 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.253234    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:09:17.255837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.255812    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg" event={"ID":"36e8a420-a282-4900-8b0f-db0c7f264a25","Type":"ContainerStarted","Data":"7e7e7fdb58af0b0e38c87680c236cdd49adeb90811cfcb8a692f767f5ccaa053"}
Sep 16 13:09:17.257809 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.257747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerStarted","Data":"0a2639c0fb3338ecfb608c2817171bfffac668e007cf878bbacf559cd8a06843"}
Sep 16 13:09:17.272432 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.272408    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:09:17.273163 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.273145    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:09:17.280847 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.280806    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq" podStartSLOduration=6.834534108 podStartE2EDuration="27.280794135s" totalImagesPullingTime="20.446260027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:17.279975583 +0000 UTC m=+28.702717161" watchObservedRunningTime="2026-09-16 13:09:17.280794135 +0000 UTC m=+28.703535714"
Sep 16 13:09:17.320661 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:17.320611    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2d6mg" podStartSLOduration=1.878504398 podStartE2EDuration="27.320595612s" totalImagesPullingTime="25.442091214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:17.320538054 +0000 UTC m=+28.743279632" watchObservedRunningTime="2026-09-16 13:09:17.320595612 +0000 UTC m=+28.743337190"
Sep 16 13:09:18.102928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.102899    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:18.103220 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.102899    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:18.103220 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:18.103000    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:18.103220 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:18.103065    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:18.262338 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.262290    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e4d7bb-a82f-4811-8a44-0c3ac78869b7" containerID="0a2639c0fb3338ecfb608c2817171bfffac668e007cf878bbacf559cd8a06843" exitCode=0
Sep 16 13:09:18.262707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.262431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerDied","Data":"0a2639c0fb3338ecfb608c2817171bfffac668e007cf878bbacf559cd8a06843"}
Sep 16 13:09:18.758427 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.758398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dpv4k"]
Sep 16 13:09:18.758597 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.758515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:18.758658 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:18.758634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:18.761691 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.761656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vf2vf"]
Sep 16 13:09:18.761846 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.761764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:18.761922 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:18.761904    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:18.762497 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.762429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vgrv9"]
Sep 16 13:09:18.762593 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:18.762531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:18.762633 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:18.762600    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:20.103650 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:20.103549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:20.103650 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:20.103587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:20.104348 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:20.103652    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:20.104348 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:20.103707    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:20.269441 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:20.269412    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e4d7bb-a82f-4811-8a44-0c3ac78869b7" containerID="78233652e62f99833a552c140e85cbc168226cbb8f0c099ee5e31ed99e2019be" exitCode=0
Sep 16 13:09:20.269634 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:20.269463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerDied","Data":"78233652e62f99833a552c140e85cbc168226cbb8f0c099ee5e31ed99e2019be"}
Sep 16 13:09:21.103010 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:21.102944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:21.103152 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.103053    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:21.826764 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:21.826738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:21.827096 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:21.826788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:21.827096 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.826883    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:09:21.827096 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.826885    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:09:21.827096 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.826929    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret podName:5111c756-aa34-4811-a126-504b86fac9d3 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:53.826916399 +0000 UTC m=+65.249657955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret") pod "global-pull-secret-syncer-vf2vf" (UID: "5111c756-aa34-4811-a126-504b86fac9d3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:09:21.827096 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.826941    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs podName:19312944-1bba-4df0-b850-4849fbb74797 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:53.826935727 +0000 UTC m=+65.249677284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs") pod "network-metrics-daemon-dpv4k" (UID: "19312944-1bba-4df0-b850-4849fbb74797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:09:21.927729 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:21.927705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:21.927872 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.927829    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:09:21.927872 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.927843    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:09:21.927872 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.927866    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ngfg5 for pod openshift-network-diagnostics/network-check-target-vgrv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:09:21.927986 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:21.927913    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5 podName:86f38672-4e25-4e3e-bc7c-e2aee1294de1 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:53.927899799 +0000 UTC m=+65.350641355 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngfg5" (UniqueName: "kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5") pod "network-check-target-vgrv9" (UID: "86f38672-4e25-4e3e-bc7c-e2aee1294de1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:09:22.103580 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:22.103501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:22.103733 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:22.103516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:22.103733 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:22.103629    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:22.103733 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:22.103699    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:22.276047 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:22.276013    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e4d7bb-a82f-4811-8a44-0c3ac78869b7" containerID="8f56ff3466f649845caad728b94bb12f1bde971d7333c1e6065f1708f0fa08ed" exitCode=0
Sep 16 13:09:22.276210 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:22.276074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerDied","Data":"8f56ff3466f649845caad728b94bb12f1bde971d7333c1e6065f1708f0fa08ed"}
Sep 16 13:09:23.107648 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:23.107618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:23.108197 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:23.107744    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:24.103437 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:24.103353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:24.103594 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:24.103470    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:24.103594 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:24.103525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:24.103672 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:24.103610    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:25.103133 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:25.103098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:25.103482 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:25.103209    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:26.103402 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:26.103366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:26.103834 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:26.103499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:26.103834 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:26.103588    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:26.103834 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:26.103494    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:27.103562 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:27.103534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:27.103980 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:27.103651    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:28.103183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:28.103108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:28.103305 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:28.103213    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:28.103305 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:28.103267    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:28.103374 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:28.103353    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:29.106969 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:29.105002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:29.106969 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:29.105224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:29.295767 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:29.295735    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e4d7bb-a82f-4811-8a44-0c3ac78869b7" containerID="4ced6578a4f2ef8dd40f1faf57bb6c6dd9c255b475635e1c3612613db7ea8442" exitCode=0
Sep 16 13:09:29.295925 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:29.295778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerDied","Data":"4ced6578a4f2ef8dd40f1faf57bb6c6dd9c255b475635e1c3612613db7ea8442"}
Sep 16 13:09:30.103072 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:30.103039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:30.103309 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:30.103146    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:30.103309 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:30.103209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:30.103430 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:30.103322    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:30.301507 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:30.301473    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e4d7bb-a82f-4811-8a44-0c3ac78869b7" containerID="13780c1d0472b67988427430eb7d2f22424210c9560fb99b6ab9426c7dffb4fe" exitCode=0
Sep 16 13:09:30.301912 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:30.301537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerDied","Data":"13780c1d0472b67988427430eb7d2f22424210c9560fb99b6ab9426c7dffb4fe"}
Sep 16 13:09:31.103160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:31.103124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:31.103354 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:31.103229    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:31.307008 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:31.306970    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htcdk" event={"ID":"a7e4d7bb-a82f-4811-8a44-0c3ac78869b7","Type":"ContainerStarted","Data":"ea198bf4abdb8143727fe36a36178aa6fd1a97fdbafbaf41a6d62f826252986b"}
Sep 16 13:09:31.332237 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:31.330165    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-htcdk" podStartSLOduration=14.001945981 podStartE2EDuration="41.330145368s" totalImagesPullingTime="27.328199387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:31.330032089 +0000 UTC m=+42.752773671" watchObservedRunningTime="2026-09-16 13:09:31.330145368 +0000 UTC m=+42.752886945"
Sep 16 13:09:32.103726 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:32.103691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:32.103921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:32.103730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:32.103921 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:32.103788    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:32.104005 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:32.103929    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:33.103541 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:33.103508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:33.103964 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:33.103634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:34.103441 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:34.103393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:34.103441 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:34.103445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:34.103916 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:34.103514    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:34.103916 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:34.103661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:35.103678 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:35.103645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:35.104082 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:35.103754    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:36.103455 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:36.103372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:36.103604 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:36.103372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:36.103604 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:36.103481    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:36.103604 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:36.103569    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:37.103650 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:37.103618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:37.104071 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:37.103721    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:38.103039 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:38.103005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:38.103251 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:38.103045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:38.103251 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:38.103144    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:38.103251 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:38.103231    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:39.107430 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:39.107398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:39.107887 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:39.107499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:40.103069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:40.103037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:40.103263 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:40.103149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:40.103263 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:40.103198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:40.103359 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:40.103290    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:41.103165 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:41.103126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:41.103631 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:41.103239    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:42.103460 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:42.103428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:42.103879 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:42.103435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:42.103879 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:42.103535    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:42.103879 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:42.103612    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:43.102959 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:43.102928    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:43.103197 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:43.103033    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:44.103004 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:44.102970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:44.103378 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:44.102970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:44.103378 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:44.103082    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:44.103378 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:44.103143    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:45.103293 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:45.103258    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:45.103767 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:45.103362    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vf2vf" podUID="5111c756-aa34-4811-a126-504b86fac9d3"
Sep 16 13:09:46.103819 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.103781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:46.104199 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:46.103906    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpv4k" podUID="19312944-1bba-4df0-b850-4849fbb74797"
Sep 16 13:09:46.104199 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.103963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:46.104199 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:46.104046    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgrv9" podUID="86f38672-4e25-4e3e-bc7c-e2aee1294de1"
Sep 16 13:09:46.802019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.801986    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeReady"
Sep 16 13:09:46.802184 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.802132    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 13:09:46.854628 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.854582    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r5lhr"]
Sep 16 13:09:46.866608 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.866581    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wqf4x"]
Sep 16 13:09:46.866744 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.866705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:46.870759 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.870734    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 13:09:46.870917 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.870765    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftxst\""
Sep 16 13:09:46.871665 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.871649    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 13:09:46.877726 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.877707    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r5lhr"]
Sep 16 13:09:46.877821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.877736    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqf4x"]
Sep 16 13:09:46.877902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.877820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:46.880893 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.880869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dlvxs\""
Sep 16 13:09:46.880985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.880876    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 13:09:46.880985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.880925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 13:09:46.880985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.880945    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 13:09:46.880985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.880954    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 13:09:46.906100 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38ca1309-d752-41bd-ab24-91703d4ee35e-metrics-tls\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:46.906189 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38ca1309-d752-41bd-ab24-91703d4ee35e-config-volume\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:46.906189 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:46.906189 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-crio-socket\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:46.906189 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:46.906380 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgkv\" (UniqueName: \"kubernetes.io/projected/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-kube-api-access-lwgkv\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:46.906380 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-data-volume\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:46.906380 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ca1309-d752-41bd-ab24-91703d4ee35e-tmp-dir\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:46.906380 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.906342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtl89\" (UniqueName: \"kubernetes.io/projected/38ca1309-d752-41bd-ab24-91703d4ee35e-kube-api-access-jtl89\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:46.956330 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.956300    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mgxfc"]
Sep 16 13:09:46.966949 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.966930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mgxfc"
Sep 16 13:09:46.969185 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.969160    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 13:09:46.969305 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.969240    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dbtv4\""
Sep 16 13:09:46.969305 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.969264    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 13:09:46.970792 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.970775    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 13:09:46.971270 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:46.971252    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mgxfc"]
Sep 16 13:09:47.007085 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-data-volume\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.007211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ca1309-d752-41bd-ab24-91703d4ee35e-tmp-dir\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.007211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtl89\" (UniqueName: \"kubernetes.io/projected/38ca1309-d752-41bd-ab24-91703d4ee35e-kube-api-access-jtl89\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.007211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38ca1309-d752-41bd-ab24-91703d4ee35e-metrics-tls\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.007211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38ca1309-d752-41bd-ab24-91703d4ee35e-config-volume\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.007211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.007478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-crio-socket\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.007478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/456f9723-c1e7-46f3-9746-aeccf33c3271-cert\") pod \"ingress-canary-mgxfc\" (UID: \"456f9723-c1e7-46f3-9746-aeccf33c3271\") " pod="openshift-ingress-canary/ingress-canary-mgxfc"
Sep 16 13:09:47.007478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.007478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgkv\" (UniqueName: \"kubernetes.io/projected/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-kube-api-access-lwgkv\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.007478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m258b\" (UniqueName: \"kubernetes.io/projected/456f9723-c1e7-46f3-9746-aeccf33c3271-kube-api-access-m258b\") pod \"ingress-canary-mgxfc\" (UID: \"456f9723-c1e7-46f3-9746-aeccf33c3271\") " pod="openshift-ingress-canary/ingress-canary-mgxfc"
Sep 16 13:09:47.007478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ca1309-d752-41bd-ab24-91703d4ee35e-tmp-dir\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.007478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-data-volume\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.007718 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-crio-socket\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.007718 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38ca1309-d752-41bd-ab24-91703d4ee35e-config-volume\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.007920 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.007894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.011070 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.011047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38ca1309-d752-41bd-ab24-91703d4ee35e-metrics-tls\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.011070 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.011058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.018945 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.018917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtl89\" (UniqueName: \"kubernetes.io/projected/38ca1309-d752-41bd-ab24-91703d4ee35e-kube-api-access-jtl89\") pod \"dns-default-r5lhr\" (UID: \"38ca1309-d752-41bd-ab24-91703d4ee35e\") " pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.019128 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.019106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgkv\" (UniqueName: \"kubernetes.io/projected/b2242fad-8a9b-43bc-a5bb-1ea4508bae38-kube-api-access-lwgkv\") pod \"insights-runtime-extractor-wqf4x\" (UID: \"b2242fad-8a9b-43bc-a5bb-1ea4508bae38\") " pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.103183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.103095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:47.105473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.105452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 13:09:47.108590 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.108566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m258b\" (UniqueName: \"kubernetes.io/projected/456f9723-c1e7-46f3-9746-aeccf33c3271-kube-api-access-m258b\") pod \"ingress-canary-mgxfc\" (UID: \"456f9723-c1e7-46f3-9746-aeccf33c3271\") " pod="openshift-ingress-canary/ingress-canary-mgxfc"
Sep 16 13:09:47.108683 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.108672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/456f9723-c1e7-46f3-9746-aeccf33c3271-cert\") pod \"ingress-canary-mgxfc\" (UID: \"456f9723-c1e7-46f3-9746-aeccf33c3271\") " pod="openshift-ingress-canary/ingress-canary-mgxfc"
Sep 16 13:09:47.111062 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.111043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/456f9723-c1e7-46f3-9746-aeccf33c3271-cert\") pod \"ingress-canary-mgxfc\" (UID: \"456f9723-c1e7-46f3-9746-aeccf33c3271\") " pod="openshift-ingress-canary/ingress-canary-mgxfc"
Sep 16 13:09:47.116419 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.116401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m258b\" (UniqueName: \"kubernetes.io/projected/456f9723-c1e7-46f3-9746-aeccf33c3271-kube-api-access-m258b\") pod \"ingress-canary-mgxfc\" (UID: \"456f9723-c1e7-46f3-9746-aeccf33c3271\") " pod="openshift-ingress-canary/ingress-canary-mgxfc"
Sep 16 13:09:47.176697 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.176666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:47.185801 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.185775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqf4x"
Sep 16 13:09:47.275729 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.275700    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mgxfc"
Sep 16 13:09:47.466921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.466877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqf4x"]
Sep 16 13:09:47.469693 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.469667    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r5lhr"]
Sep 16 13:09:47.471266 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:47.471242    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2242fad_8a9b_43bc_a5bb_1ea4508bae38.slice/crio-d021bcf07deef214abc998021bdc2790d45345214084c0fb51732922dbcd8b5b WatchSource:0}: Error finding container d021bcf07deef214abc998021bdc2790d45345214084c0fb51732922dbcd8b5b: Status 404 returned error can't find the container with id d021bcf07deef214abc998021bdc2790d45345214084c0fb51732922dbcd8b5b
Sep 16 13:09:47.472521 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:47.472502    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ca1309_d752_41bd_ab24_91703d4ee35e.slice/crio-982bc4032b949dc766a4f12be695961914286bd1acdafa01c408a634f9c99a12 WatchSource:0}: Error finding container 982bc4032b949dc766a4f12be695961914286bd1acdafa01c408a634f9c99a12: Status 404 returned error can't find the container with id 982bc4032b949dc766a4f12be695961914286bd1acdafa01c408a634f9c99a12
Sep 16 13:09:47.546610 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:47.546545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mgxfc"]
Sep 16 13:09:47.549736 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:47.549715    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456f9723_c1e7_46f3_9746_aeccf33c3271.slice/crio-484eaf41ea4a3216a5598e9da70d9c14d389547912b66216aae55bb8f81751a9 WatchSource:0}: Error finding container 484eaf41ea4a3216a5598e9da70d9c14d389547912b66216aae55bb8f81751a9: Status 404 returned error can't find the container with id 484eaf41ea4a3216a5598e9da70d9c14d389547912b66216aae55bb8f81751a9
Sep 16 13:09:48.103679 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.103609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:48.103679 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.103638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:48.107332 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.107106    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 13:09:48.107332 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.107178    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5rcg\""
Sep 16 13:09:48.107332 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.107183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 13:09:48.107332 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.107292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sfm4x\""
Sep 16 13:09:48.107332 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.107115    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 13:09:48.347072 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.347033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mgxfc" event={"ID":"456f9723-c1e7-46f3-9746-aeccf33c3271","Type":"ContainerStarted","Data":"484eaf41ea4a3216a5598e9da70d9c14d389547912b66216aae55bb8f81751a9"}
Sep 16 13:09:48.349007 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.348970    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r5lhr" event={"ID":"38ca1309-d752-41bd-ab24-91703d4ee35e","Type":"ContainerStarted","Data":"982bc4032b949dc766a4f12be695961914286bd1acdafa01c408a634f9c99a12"}
Sep 16 13:09:48.350870 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.350830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqf4x" event={"ID":"b2242fad-8a9b-43bc-a5bb-1ea4508bae38","Type":"ContainerStarted","Data":"7c0283d43b407f9ee7dd2edc437d25c909ab1752a71ddc08e2ae623bc17efa6e"}
Sep 16 13:09:48.350981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:48.350903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqf4x" event={"ID":"b2242fad-8a9b-43bc-a5bb-1ea4508bae38","Type":"ContainerStarted","Data":"d021bcf07deef214abc998021bdc2790d45345214084c0fb51732922dbcd8b5b"}
Sep 16 13:09:49.277274 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:49.277220    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fssq"
Sep 16 13:09:50.358125 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:50.358079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqf4x" event={"ID":"b2242fad-8a9b-43bc-a5bb-1ea4508bae38","Type":"ContainerStarted","Data":"edaed21a63437a8999cecaa102955df5bb0d5629c20a51b3bb4dc3c0377ac564"}
Sep 16 13:09:50.360058 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:50.360034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r5lhr" event={"ID":"38ca1309-d752-41bd-ab24-91703d4ee35e","Type":"ContainerStarted","Data":"e9d3e7057ca5514a2a98873fe32b38498dee069513ed55f6a907ff5d95a72ef2"}
Sep 16 13:09:51.364137 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:51.364095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mgxfc" event={"ID":"456f9723-c1e7-46f3-9746-aeccf33c3271","Type":"ContainerStarted","Data":"2dcd87a5644ed5f320720cc68b6076600b777eea7ecc5d365f6f2efec37ab5ae"}
Sep 16 13:09:51.366218 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:51.366194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r5lhr" event={"ID":"38ca1309-d752-41bd-ab24-91703d4ee35e","Type":"ContainerStarted","Data":"58309e4811ea0e482e355e96f66e4af39b71391d9958d39334ad59ff51c03e58"}
Sep 16 13:09:51.366357 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:51.366315    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:09:51.384350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:51.384281    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mgxfc" podStartSLOduration=2.179399424 podStartE2EDuration="5.384262109s" totalImagesPullingTime="3.204862685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:51.383709311 +0000 UTC m=+62.806450903" watchObservedRunningTime="2026-09-16 13:09:51.384262109 +0000 UTC m=+62.807003684"
Sep 16 13:09:51.404573 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:51.404516    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r5lhr" podStartSLOduration=3.292537181 podStartE2EDuration="5.404494607s" totalImagesPullingTime="2.111957426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:51.403959657 +0000 UTC m=+62.826701237" watchObservedRunningTime="2026-09-16 13:09:51.404494607 +0000 UTC m=+62.827236188"
Sep 16 13:09:52.372453 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:52.372357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqf4x" event={"ID":"b2242fad-8a9b-43bc-a5bb-1ea4508bae38","Type":"ContainerStarted","Data":"14bd57ec36fcf81a8a737d9d84b090fe192f573eb3f126b1e72d3187bddc2852"}
Sep 16 13:09:52.392606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:52.390341    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wqf4x" podStartSLOduration=2.153004049 podStartE2EDuration="6.390313075s" totalImagesPullingTime="4.237309026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:09:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:52.389133298 +0000 UTC m=+63.811874879" watchObservedRunningTime="2026-09-16 13:09:52.390313075 +0000 UTC m=+63.813054655"
Sep 16 13:09:53.337652 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.337615    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"]
Sep 16 13:09:53.350259 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.350233    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"]
Sep 16 13:09:53.350394 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.350327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.353424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.353394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 13:09:53.353540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.353431    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 13:09:53.353540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.353434    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 13:09:53.353540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.353467    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-s2jlr\""
Sep 16 13:09:53.353752 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.353736    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 13:09:53.353921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.353904    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 13:09:53.457083 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.457052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.457083 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.457081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.457467 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.457269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.457467 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.457332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bgg\" (UniqueName: \"kubernetes.io/projected/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-kube-api-access-p5bgg\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.558520 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.558483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bgg\" (UniqueName: \"kubernetes.io/projected/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-kube-api-access-p5bgg\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.558672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.558550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.558672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.558579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.558672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.558644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.559305 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.559279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.560956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.560934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.561064 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.560956    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.566097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.566076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bgg\" (UniqueName: \"kubernetes.io/projected/5e441b49-17b3-4e4c-92e2-d263ea8cf8ff-kube-api-access-p5bgg\") pod \"prometheus-operator-68bcd7f987-4j2gm\" (UID: \"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.659166 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.659093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"
Sep 16 13:09:53.860705 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.860656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:53.860705 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.860716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:53.863092 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.863074    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 13:09:53.863144 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.863074    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 13:09:53.873942 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.873920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5111c756-aa34-4811-a126-504b86fac9d3-original-pull-secret\") pod \"global-pull-secret-syncer-vf2vf\" (UID: \"5111c756-aa34-4811-a126-504b86fac9d3\") " pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:53.874022 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.873987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19312944-1bba-4df0-b850-4849fbb74797-metrics-certs\") pod \"network-metrics-daemon-dpv4k\" (UID: \"19312944-1bba-4df0-b850-4849fbb74797\") " pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:53.947814 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.947784    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm"]
Sep 16 13:09:53.951009 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:53.950985    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e441b49_17b3_4e4c_92e2_d263ea8cf8ff.slice/crio-a0c65e047f77c04bac09ba8a053af6c7d6546dd06a0afa9ce619c055e092c82d WatchSource:0}: Error finding container a0c65e047f77c04bac09ba8a053af6c7d6546dd06a0afa9ce619c055e092c82d: Status 404 returned error can't find the container with id a0c65e047f77c04bac09ba8a053af6c7d6546dd06a0afa9ce619c055e092c82d
Sep 16 13:09:53.961057 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.961032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:53.963014 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.962992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 13:09:53.973287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.973266    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 13:09:53.984156 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:53.984133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfg5\" (UniqueName: \"kubernetes.io/projected/86f38672-4e25-4e3e-bc7c-e2aee1294de1-kube-api-access-ngfg5\") pod \"network-check-target-vgrv9\" (UID: \"86f38672-4e25-4e3e-bc7c-e2aee1294de1\") " pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:54.011605 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.011581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vf2vf"
Sep 16 13:09:54.119807 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.119780    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sfm4x\""
Sep 16 13:09:54.125098 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.125080    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5rcg\""
Sep 16 13:09:54.127807 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.127794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:09:54.134349 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.134327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpv4k"
Sep 16 13:09:54.275818 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.275784    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vf2vf"]
Sep 16 13:09:54.278918 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:54.278891    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5111c756_aa34_4811_a126_504b86fac9d3.slice/crio-0a68d3e2ec30759580020680386361d7b6ad1c688b3f2d37c7ed4e29300a52ef WatchSource:0}: Error finding container 0a68d3e2ec30759580020680386361d7b6ad1c688b3f2d37c7ed4e29300a52ef: Status 404 returned error can't find the container with id 0a68d3e2ec30759580020680386361d7b6ad1c688b3f2d37c7ed4e29300a52ef
Sep 16 13:09:54.380978 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.380943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vf2vf" event={"ID":"5111c756-aa34-4811-a126-504b86fac9d3","Type":"ContainerStarted","Data":"0a68d3e2ec30759580020680386361d7b6ad1c688b3f2d37c7ed4e29300a52ef"}
Sep 16 13:09:54.382181 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.382155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm" event={"ID":"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff","Type":"ContainerStarted","Data":"a0c65e047f77c04bac09ba8a053af6c7d6546dd06a0afa9ce619c055e092c82d"}
Sep 16 13:09:54.396393 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.396371    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vgrv9"]
Sep 16 13:09:54.399616 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:54.399590    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f38672_4e25_4e3e_bc7c_e2aee1294de1.slice/crio-2eae085d8bfc7bbd2481f4a4b55a26e0e58c147d0bbb84fef26c8230fff8bd1c WatchSource:0}: Error finding container 2eae085d8bfc7bbd2481f4a4b55a26e0e58c147d0bbb84fef26c8230fff8bd1c: Status 404 returned error can't find the container with id 2eae085d8bfc7bbd2481f4a4b55a26e0e58c147d0bbb84fef26c8230fff8bd1c
Sep 16 13:09:54.415948 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:54.415924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dpv4k"]
Sep 16 13:09:54.418925 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:54.418895    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19312944_1bba_4df0_b850_4849fbb74797.slice/crio-e13df5960a680c8a1552eadd8b8e9357163d299cc200f8688a745cd7b7859a8f WatchSource:0}: Error finding container e13df5960a680c8a1552eadd8b8e9357163d299cc200f8688a745cd7b7859a8f: Status 404 returned error can't find the container with id e13df5960a680c8a1552eadd8b8e9357163d299cc200f8688a745cd7b7859a8f
Sep 16 13:09:55.394311 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:55.393775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpv4k" event={"ID":"19312944-1bba-4df0-b850-4849fbb74797","Type":"ContainerStarted","Data":"e13df5960a680c8a1552eadd8b8e9357163d299cc200f8688a745cd7b7859a8f"}
Sep 16 13:09:55.396797 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:55.396451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vgrv9" event={"ID":"86f38672-4e25-4e3e-bc7c-e2aee1294de1","Type":"ContainerStarted","Data":"2eae085d8bfc7bbd2481f4a4b55a26e0e58c147d0bbb84fef26c8230fff8bd1c"}
Sep 16 13:09:56.403357 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:56.403318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpv4k" event={"ID":"19312944-1bba-4df0-b850-4849fbb74797","Type":"ContainerStarted","Data":"0ea254aed25e33d1131aab039122fd2d5919194215b610dbb5a096719c59f5a0"}
Sep 16 13:09:56.403821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:56.403386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpv4k" event={"ID":"19312944-1bba-4df0-b850-4849fbb74797","Type":"ContainerStarted","Data":"c69e786ce66df86005ed83fb6f4bf02009cac5999b92d8bf3b12e0a2680c4bf8"}
Sep 16 13:09:56.407242 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:56.406990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm" event={"ID":"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff","Type":"ContainerStarted","Data":"ff77bb18562eaeb3cf15499d2ffc9d7a84ccf17d2a93948f77f761f9d896ae98"}
Sep 16 13:09:56.407242 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:56.407046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm" event={"ID":"5e441b49-17b3-4e4c-92e2-d263ea8cf8ff","Type":"ContainerStarted","Data":"494fab772c9884e715dd7cfe41b571cacd444e6067451b04bb156d010a3b5eb3"}
Sep 16 13:09:56.422685 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:56.422610    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dpv4k" podStartSLOduration=64.93380842 podStartE2EDuration="1m6.422589503s" totalImagesPullingTime="1.488781083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:56.421038016 +0000 UTC m=+67.843779607" watchObservedRunningTime="2026-09-16 13:09:56.422589503 +0000 UTC m=+67.845331082"
Sep 16 13:09:56.439072 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:56.439014    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j2gm" podStartSLOduration=1.481033124 podStartE2EDuration="3.438993245s" totalImagesPullingTime="1.957960121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:56.437777557 +0000 UTC m=+67.860519171" watchObservedRunningTime="2026-09-16 13:09:56.438993245 +0000 UTC m=+67.861734822"
Sep 16 13:09:58.679814 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.679771    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"]
Sep 16 13:09:58.682141 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.682122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.689095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.689069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 13:09:58.689229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.689119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 13:09:58.689486 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.689394    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kb4t6\""
Sep 16 13:09:58.693922 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.693903    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"]
Sep 16 13:09:58.696424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.696402    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"]
Sep 16 13:09:58.698743 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.698724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.701104 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.701083    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 13:09:58.701199 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.701119    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 13:09:58.701199 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.701090    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 13:09:58.701304 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.701204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gtfd9\""
Sep 16 13:09:58.715621 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.715599    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"]
Sep 16 13:09:58.722737 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.722711    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ndhbc"]
Sep 16 13:09:58.725369 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.725349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.727672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.727650    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-djg9t\""
Sep 16 13:09:58.727938 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.727919    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 13:09:58.728500 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.728303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 13:09:58.728591 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.728449    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 13:09:58.795113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.795113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.795401 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.795401 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-wtmp\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.795401 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.795401 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6f7\" (UniqueName: \"kubernetes.io/projected/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-api-access-nc6f7\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.795401 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.795401 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.795401 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-textfile\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f39f477-f6c7-4b47-a65d-1810120be64c-metrics-client-ca\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-root\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4qd\" (UniqueName: \"kubernetes.io/projected/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-kube-api-access-8d4qd\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-sys\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-tls\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.795876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.795872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfg6s\" (UniqueName: \"kubernetes.io/projected/9f39f477-f6c7-4b47-a65d-1810120be64c-kube-api-access-lfg6s\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.896527 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.896711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.896711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-sys\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.896711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-tls\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.896711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.896711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfg6s\" (UniqueName: \"kubernetes.io/projected/9f39f477-f6c7-4b47-a65d-1810120be64c-kube-api-access-lfg6s\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.896711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-sys\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:58.896749    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:09:58.896813    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-tls podName:9f39f477-f6c7-4b47-a65d-1810120be64c nodeName:}" failed. No retries permitted until 2026-09-16 13:09:59.396790324 +0000 UTC m=+70.819531882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-tls") pod "node-exporter-ndhbc" (UID: "9f39f477-f6c7-4b47-a65d-1810120be64c") : secret "node-exporter-tls" not found
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-wtmp\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6f7\" (UniqueName: \"kubernetes.io/projected/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-api-access-nc6f7\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.896987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-textfile\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f39f477-f6c7-4b47-a65d-1810120be64c-metrics-client-ca\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-root\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4qd\" (UniqueName: \"kubernetes.io/projected/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-kube-api-access-8d4qd\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.898703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.899510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.899510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.899510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-wtmp\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.899510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-textfile\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.899510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f39f477-f6c7-4b47-a65d-1810120be64c-root\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.899510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.897968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.899510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.898103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.899510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.898431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f39f477-f6c7-4b47-a65d-1810120be64c-metrics-client-ca\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.899924 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.899618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.900522 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.900500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.900795 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.900769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.900906 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.900794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.900906 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.900819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.909694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.909675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfg6s\" (UniqueName: \"kubernetes.io/projected/9f39f477-f6c7-4b47-a65d-1810120be64c-kube-api-access-lfg6s\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:58.913145 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.913102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6f7\" (UniqueName: \"kubernetes.io/projected/8ff8fbe5-75a8-4cc4-a468-d2c660ce6679-kube-api-access-nc6f7\") pod \"kube-state-metrics-59fbdd77d-m77xn\" (UID: \"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:58.913145 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.913123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4qd\" (UniqueName: \"kubernetes.io/projected/fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657-kube-api-access-8d4qd\") pod \"openshift-state-metrics-67dcf8c979-jt9mv\" (UID: \"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:58.993225 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:58.993188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"
Sep 16 13:09:59.008985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.008958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"
Sep 16 13:09:59.401977 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.401881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-tls\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:59.403177 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.403154    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-c4vg9"]
Sep 16 13:09:59.404240 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.404214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f39f477-f6c7-4b47-a65d-1810120be64c-node-exporter-tls\") pod \"node-exporter-ndhbc\" (UID: \"9f39f477-f6c7-4b47-a65d-1810120be64c\") " pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:59.422035 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.422013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-c4vg9"]
Sep 16 13:09:59.422150 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.422098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-c4vg9"
Sep 16 13:09:59.424152 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.424130    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 13:09:59.424263 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.424161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ww2jv\""
Sep 16 13:09:59.424393 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.424378    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 13:09:59.503219 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.503188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkjr8\" (UniqueName: \"kubernetes.io/projected/78bf4632-a381-4760-bec8-3b52200112ac-kube-api-access-qkjr8\") pod \"downloads-76f99584f4-c4vg9\" (UID: \"78bf4632-a381-4760-bec8-3b52200112ac\") " pod="openshift-console/downloads-76f99584f4-c4vg9"
Sep 16 13:09:59.604421 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.604386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkjr8\" (UniqueName: \"kubernetes.io/projected/78bf4632-a381-4760-bec8-3b52200112ac-kube-api-access-qkjr8\") pod \"downloads-76f99584f4-c4vg9\" (UID: \"78bf4632-a381-4760-bec8-3b52200112ac\") " pod="openshift-console/downloads-76f99584f4-c4vg9"
Sep 16 13:09:59.615508 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.615462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkjr8\" (UniqueName: \"kubernetes.io/projected/78bf4632-a381-4760-bec8-3b52200112ac-kube-api-access-qkjr8\") pod \"downloads-76f99584f4-c4vg9\" (UID: \"78bf4632-a381-4760-bec8-3b52200112ac\") " pod="openshift-console/downloads-76f99584f4-c4vg9"
Sep 16 13:09:59.634739 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.634695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ndhbc"
Sep 16 13:09:59.658800 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:59.658773    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f39f477_f6c7_4b47_a65d_1810120be64c.slice/crio-b3cd9575ef07d25f70677377211d7f99c3ae8ce019a0919c1ef9f2a4b4238163 WatchSource:0}: Error finding container b3cd9575ef07d25f70677377211d7f99c3ae8ce019a0919c1ef9f2a4b4238163: Status 404 returned error can't find the container with id b3cd9575ef07d25f70677377211d7f99c3ae8ce019a0919c1ef9f2a4b4238163
Sep 16 13:09:59.731663 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.731629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-c4vg9"
Sep 16 13:09:59.779020 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.778912    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv"]
Sep 16 13:09:59.782156 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.782044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn"]
Sep 16 13:09:59.783780 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:59.783747    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6ddea4_1d1c_4a7c_a1a5_cdb02b122657.slice/crio-f183c90f2182e2c07d7b444023fb6cf54d71c669c420c3e73853d5ea83abf9e2 WatchSource:0}: Error finding container f183c90f2182e2c07d7b444023fb6cf54d71c669c420c3e73853d5ea83abf9e2: Status 404 returned error can't find the container with id f183c90f2182e2c07d7b444023fb6cf54d71c669c420c3e73853d5ea83abf9e2
Sep 16 13:09:59.787978 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:09:59.787825    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff8fbe5_75a8_4cc4_a468_d2c660ce6679.slice/crio-7bda97d1fdda84feff9fa19339a111838c0894a156d63cf96b7243cab723b966 WatchSource:0}: Error finding container 7bda97d1fdda84feff9fa19339a111838c0894a156d63cf96b7243cab723b966: Status 404 returned error can't find the container with id 7bda97d1fdda84feff9fa19339a111838c0894a156d63cf96b7243cab723b966
Sep 16 13:09:59.834067 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.834037    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:09:59.860766 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.860735    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:09:59.860941 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.860831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.863245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.863228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 13:09:59.863356 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.863253    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 13:09:59.863459 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.863444    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r9j2b\""
Sep 16 13:09:59.863775 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.863758    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 13:09:59.863914 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.863898    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 13:09:59.863978 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.863942    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 13:09:59.864039 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.864026    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 13:09:59.864093 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.864030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 13:09:59.864226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.864206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 13:09:59.864277 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.864228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 13:09:59.907928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.907868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.907928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.907914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908086 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.907943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908086 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.907970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsd9p\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-kube-api-access-gsd9p\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908086 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908236 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908236 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908236 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908236 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908422 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-out\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908422 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-volume\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908422 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-web-config\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:09:59.908422 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:09:59.908353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009092 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-out\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009261 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-volume\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009261 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-web-config\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009261 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009261 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009261 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009261 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009574 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsd9p\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-kube-api-access-gsd9p\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009574 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009574 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009574 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009574 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009574 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.009872 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.009772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.011215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.010553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.011215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.011132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.012502 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.012438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.013095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.013068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.013250 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.013228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.013320 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.013278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-out\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.013381 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.013352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-volume\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.013573 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.013550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.013701 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.013678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-web-config\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.014011 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.013991    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.014183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.014164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.017821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.017803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsd9p\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-kube-api-access-gsd9p\") pod \"alertmanager-main-0\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.025044 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.025022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-c4vg9"]
Sep 16 13:10:00.027963 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:10:00.027941    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bf4632_a381_4760_bec8_3b52200112ac.slice/crio-ad40654c0cd2c280fd22d49177237d7aa8f08ae5c2b531ddfa3170ffc585b662 WatchSource:0}: Error finding container ad40654c0cd2c280fd22d49177237d7aa8f08ae5c2b531ddfa3170ffc585b662: Status 404 returned error can't find the container with id ad40654c0cd2c280fd22d49177237d7aa8f08ae5c2b531ddfa3170ffc585b662
Sep 16 13:10:00.177421 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.177338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:10:00.425632 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.425422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vgrv9" event={"ID":"86f38672-4e25-4e3e-bc7c-e2aee1294de1","Type":"ContainerStarted","Data":"a811b98d478843fcba521ce9cf40d903bfcf2579db705c46ef30ad0854163f95"}
Sep 16 13:10:00.425632 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.425520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:10:00.427712 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.427641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv" event={"ID":"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657","Type":"ContainerStarted","Data":"8cd5224b5132930d180edad8c56d58bd865f75a26cb6354af68fcaecb13daaea"}
Sep 16 13:10:00.427712 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.427702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv" event={"ID":"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657","Type":"ContainerStarted","Data":"ac7253ef60a6c6a6abb10731c55edce20465f180da31a9bf1566179a6084ccc9"}
Sep 16 13:10:00.428017 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.427723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv" event={"ID":"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657","Type":"ContainerStarted","Data":"f183c90f2182e2c07d7b444023fb6cf54d71c669c420c3e73853d5ea83abf9e2"}
Sep 16 13:10:00.429069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.429043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-c4vg9" event={"ID":"78bf4632-a381-4760-bec8-3b52200112ac","Type":"ContainerStarted","Data":"ad40654c0cd2c280fd22d49177237d7aa8f08ae5c2b531ddfa3170ffc585b662"}
Sep 16 13:10:00.430724 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.430694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vf2vf" event={"ID":"5111c756-aa34-4811-a126-504b86fac9d3","Type":"ContainerStarted","Data":"92cc52260f25e6c5bb58e41f4ff5248e8e4e96be930766d4565e4d2b3cad9d1a"}
Sep 16 13:10:00.432096 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.432070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndhbc" event={"ID":"9f39f477-f6c7-4b47-a65d-1810120be64c","Type":"ContainerStarted","Data":"b3cd9575ef07d25f70677377211d7f99c3ae8ce019a0919c1ef9f2a4b4238163"}
Sep 16 13:10:00.433335 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.433318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn" event={"ID":"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679","Type":"ContainerStarted","Data":"7bda97d1fdda84feff9fa19339a111838c0894a156d63cf96b7243cab723b966"}
Sep 16 13:10:00.441795 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.441753    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vgrv9" podStartSLOduration=65.334076735 podStartE2EDuration="1m10.441737283s" totalImagesPullingTime="5.107660548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:00.439924722 +0000 UTC m=+71.862666303" watchObservedRunningTime="2026-09-16 13:10:00.441737283 +0000 UTC m=+71.864478863"
Sep 16 13:10:00.454240 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.454190    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vf2vf" podStartSLOduration=65.215376592 podStartE2EDuration="1m10.454165045s" totalImagesPullingTime="5.238788453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:00.453905889 +0000 UTC m=+71.876647470" watchObservedRunningTime="2026-09-16 13:10:00.454165045 +0000 UTC m=+71.876906618"
Sep 16 13:10:00.487071 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:00.487038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:10:00.488084 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:10:00.488042    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bdd8f3_2aea_448f_8214_e7ed9e38db29.slice/crio-ce1f99c08dd9299a3c634b136ff8c6627a7dfba950f2530780cd8a6e6fd2079b WatchSource:0}: Error finding container ce1f99c08dd9299a3c634b136ff8c6627a7dfba950f2530780cd8a6e6fd2079b: Status 404 returned error can't find the container with id ce1f99c08dd9299a3c634b136ff8c6627a7dfba950f2530780cd8a6e6fd2079b
Sep 16 13:10:01.375418 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.375377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r5lhr"
Sep 16 13:10:01.439145 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.439018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerStarted","Data":"ce1f99c08dd9299a3c634b136ff8c6627a7dfba950f2530780cd8a6e6fd2079b"}
Sep 16 13:10:01.835010 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.834967    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"]
Sep 16 13:10:01.848479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.848452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:01.852295 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.852262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 13:10:01.852434 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.852298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 13:10:01.852434 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.852367    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 13:10:01.852434 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.852262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dahdpqkrqldgt\""
Sep 16 13:10:01.852824 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.852596    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 13:10:01.852824 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.852685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 13:10:01.853346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.853078    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hq4k6\""
Sep 16 13:10:01.853940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.853918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"]
Sep 16 13:10:01.928430 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.928384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6cg9\" (UniqueName: \"kubernetes.io/projected/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-kube-api-access-s6cg9\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:01.928617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.928442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-grpc-tls\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:01.928617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.928529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:01.928740 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.928653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:01.928740 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.928692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-tls\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:01.928740 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.928735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-metrics-client-ca\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:01.928938 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.928764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:01.928938 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:01.928788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.029515 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.029459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6cg9\" (UniqueName: \"kubernetes.io/projected/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-kube-api-access-s6cg9\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.029515 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.029514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-grpc-tls\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.029954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.029557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.029954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.029645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.029954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.029697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-tls\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.029954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.029732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-metrics-client-ca\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.029954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.029765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.029954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.029792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.030895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.030866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-metrics-client-ca\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.032969 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.032839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-grpc-tls\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.032969 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.032910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-tls\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.033351 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.033307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.033608 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.033585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.034355 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.034330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.034721 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.034692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.045663 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.045639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6cg9\" (UniqueName: \"kubernetes.io/projected/52f757c9-aa31-4d10-9fcf-fe2b09a2103c-kube-api-access-s6cg9\") pod \"thanos-querier-5cf57bb6f8-f97v6\" (UID: \"52f757c9-aa31-4d10-9fcf-fe2b09a2103c\") " pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.163014 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.162927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:02.801315 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:02.801266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"]
Sep 16 13:10:02.809867 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:10:02.809827    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f757c9_aa31_4d10_9fcf_fe2b09a2103c.slice/crio-e97b7b786f193a5423057e8738f6eb0022b110a087ea22a53c23b58b4983b970 WatchSource:0}: Error finding container e97b7b786f193a5423057e8738f6eb0022b110a087ea22a53c23b58b4983b970: Status 404 returned error can't find the container with id e97b7b786f193a5423057e8738f6eb0022b110a087ea22a53c23b58b4983b970
Sep 16 13:10:03.161573 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.161483    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c856fc477-mws7l"]
Sep 16 13:10:03.164232 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.164206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.170054 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.170027    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 13:10:03.170609 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.170585    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 13:10:03.170715 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.170636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-coq0p65m4kukf\""
Sep 16 13:10:03.170715 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.170657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 13:10:03.170822 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.170711    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 13:10:03.170822 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.170655    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xrvqx\""
Sep 16 13:10:03.197474 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.197434    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c856fc477-mws7l"]
Sep 16 13:10:03.241524 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.241495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-secret-metrics-server-client-certs\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.241682 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.241537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-secret-metrics-server-tls\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.241682 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.241563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/149ded78-e698-4760-a6a8-4346af2a2db3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.241682 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.241594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-client-ca-bundle\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.241682 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.241619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/149ded78-e698-4760-a6a8-4346af2a2db3-metrics-server-audit-profiles\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.241682 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.241645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxtb\" (UniqueName: \"kubernetes.io/projected/149ded78-e698-4760-a6a8-4346af2a2db3-kube-api-access-dzxtb\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.241838 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.241724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/149ded78-e698-4760-a6a8-4346af2a2db3-audit-log\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.342663 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.342625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/149ded78-e698-4760-a6a8-4346af2a2db3-metrics-server-audit-profiles\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.342663 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.342664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxtb\" (UniqueName: \"kubernetes.io/projected/149ded78-e698-4760-a6a8-4346af2a2db3-kube-api-access-dzxtb\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.342901 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.342717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/149ded78-e698-4760-a6a8-4346af2a2db3-audit-log\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.342901 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.342752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-secret-metrics-server-client-certs\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.342901 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.342821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-secret-metrics-server-tls\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.343040 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.342903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/149ded78-e698-4760-a6a8-4346af2a2db3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.343040 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.342947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-client-ca-bundle\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.343200 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.343181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/149ded78-e698-4760-a6a8-4346af2a2db3-audit-log\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.343839 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.343813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/149ded78-e698-4760-a6a8-4346af2a2db3-metrics-server-audit-profiles\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.343990 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.343919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/149ded78-e698-4760-a6a8-4346af2a2db3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.345722 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.345694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-secret-metrics-server-tls\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.345828 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.345766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-client-ca-bundle\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.345900 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.345843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/149ded78-e698-4760-a6a8-4346af2a2db3-secret-metrics-server-client-certs\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.353177 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.353140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxtb\" (UniqueName: \"kubernetes.io/projected/149ded78-e698-4760-a6a8-4346af2a2db3-kube-api-access-dzxtb\") pod \"metrics-server-c856fc477-mws7l\" (UID: \"149ded78-e698-4760-a6a8-4346af2a2db3\") " pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.449744 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.449701    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f39f477-f6c7-4b47-a65d-1810120be64c" containerID="3e7e7825c4a233b482b8fff66683dd20c8797294ebdf0cb93b1f9c5769d220c1" exitCode=0
Sep 16 13:10:03.449942 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.449782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndhbc" event={"ID":"9f39f477-f6c7-4b47-a65d-1810120be64c","Type":"ContainerDied","Data":"3e7e7825c4a233b482b8fff66683dd20c8797294ebdf0cb93b1f9c5769d220c1"}
Sep 16 13:10:03.453120 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.452999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn" event={"ID":"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679","Type":"ContainerStarted","Data":"1782d803f917e5d02d199a8250afdb1fdef1dd70e0c9c573ea5d2e2cb4ae7c29"}
Sep 16 13:10:03.453120 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.453056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn" event={"ID":"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679","Type":"ContainerStarted","Data":"59810a393008a6df54e300d61f86d16d3c0e330f7cb418869ff0b4cb5ac30c62"}
Sep 16 13:10:03.453120 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.453086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn" event={"ID":"8ff8fbe5-75a8-4cc4-a468-d2c660ce6679","Type":"ContainerStarted","Data":"f17f2d608d05c535d3835e5f22c8b802e628cb8ac9111acb62aefcb879ff7bcd"}
Sep 16 13:10:03.454625 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.454599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6" event={"ID":"52f757c9-aa31-4d10-9fcf-fe2b09a2103c","Type":"ContainerStarted","Data":"e97b7b786f193a5423057e8738f6eb0022b110a087ea22a53c23b58b4983b970"}
Sep 16 13:10:03.457492 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.457464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv" event={"ID":"fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657","Type":"ContainerStarted","Data":"1f62d867960f20a162de8b27a8484d6d26474cfc400c0c592337a64a185bec86"}
Sep 16 13:10:03.459252 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.459225    2577 generic.go:356] "Generic (PLEG): container finished" podID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerID="c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac" exitCode=0
Sep 16 13:10:03.459318 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.459256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerDied","Data":"c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac"}
Sep 16 13:10:03.475831 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.475805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:03.553770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.552916    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"]
Sep 16 13:10:03.557028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.557006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:03.566644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.566620    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 13:10:03.570774 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.570711    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5hmdf\""
Sep 16 13:10:03.571320 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.571288    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"]
Sep 16 13:10:03.577899 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.577725    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jt9mv" podStartSLOduration=3.119592691 podStartE2EDuration="5.577705492s" totalImagesPullingTime="2.458112801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:03.577087276 +0000 UTC m=+74.999828856" watchObservedRunningTime="2026-09-16 13:10:03.577705492 +0000 UTC m=+75.000447061"
Sep 16 13:10:03.646712 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.646675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8894ffc-e25f-45b0-b5ad-d078b2f069cb-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-84mgb\" (UID: \"f8894ffc-e25f-45b0-b5ad-d078b2f069cb\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:03.646915 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.646834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9rp\" (UniqueName: \"kubernetes.io/projected/f8894ffc-e25f-45b0-b5ad-d078b2f069cb-kube-api-access-bs9rp\") pod \"monitoring-plugin-5d85778b9d-84mgb\" (UID: \"f8894ffc-e25f-45b0-b5ad-d078b2f069cb\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:03.692142 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.692077    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m77xn" podStartSLOduration=2.969049487 podStartE2EDuration="5.692054996s" totalImagesPullingTime="2.723005509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:03.690944513 +0000 UTC m=+75.113686093" watchObservedRunningTime="2026-09-16 13:10:03.692054996 +0000 UTC m=+75.114796575"
Sep 16 13:10:03.748305 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.748205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9rp\" (UniqueName: \"kubernetes.io/projected/f8894ffc-e25f-45b0-b5ad-d078b2f069cb-kube-api-access-bs9rp\") pod \"monitoring-plugin-5d85778b9d-84mgb\" (UID: \"f8894ffc-e25f-45b0-b5ad-d078b2f069cb\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:03.748464 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.748311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8894ffc-e25f-45b0-b5ad-d078b2f069cb-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-84mgb\" (UID: \"f8894ffc-e25f-45b0-b5ad-d078b2f069cb\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:03.751320 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.751290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8894ffc-e25f-45b0-b5ad-d078b2f069cb-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-84mgb\" (UID: \"f8894ffc-e25f-45b0-b5ad-d078b2f069cb\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:03.763784 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.763755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9rp\" (UniqueName: \"kubernetes.io/projected/f8894ffc-e25f-45b0-b5ad-d078b2f069cb-kube-api-access-bs9rp\") pod \"monitoring-plugin-5d85778b9d-84mgb\" (UID: \"f8894ffc-e25f-45b0-b5ad-d078b2f069cb\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:03.769797 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.769768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c856fc477-mws7l"]
Sep 16 13:10:03.773327 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:10:03.773291    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149ded78_e698_4760_a6a8_4346af2a2db3.slice/crio-7386f707dc6bbc812da58aa6ad339f3360d94477ea4c429468f6d4de6ec0c261 WatchSource:0}: Error finding container 7386f707dc6bbc812da58aa6ad339f3360d94477ea4c429468f6d4de6ec0c261: Status 404 returned error can't find the container with id 7386f707dc6bbc812da58aa6ad339f3360d94477ea4c429468f6d4de6ec0c261
Sep 16 13:10:03.879946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.879831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:03.945203 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.944081    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"]
Sep 16 13:10:03.948208 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.948168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:03.952296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.951120    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 13:10:03.952296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.951336    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 13:10:03.952296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.951582    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 13:10:03.952296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.951766    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 13:10:03.952296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.952097    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-75mhd\""
Sep 16 13:10:03.952296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.952168    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 13:10:03.966475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.966394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 13:10:03.969990 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:03.969019    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"]
Sep 16 13:10:04.051030 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.050922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-serving-certs-ca-bundle\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.051187 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.051030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.051187 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.051114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-federate-client-tls\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.051187 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.051146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-telemeter-client-tls\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.051187 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.051168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-metrics-client-ca\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.051413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.051270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-secret-telemeter-client\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.051413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.051333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.051413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.051385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tssv\" (UniqueName: \"kubernetes.io/projected/a2490c76-8ddf-4190-a31f-1de41d9142e0-kube-api-access-2tssv\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.151911 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.151875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-federate-client-tls\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.152083 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.151922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-telemeter-client-tls\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.152083 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.152005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-metrics-client-ca\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.152195 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.152088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-secret-telemeter-client\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.152195 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.152128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.152195 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.152159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tssv\" (UniqueName: \"kubernetes.io/projected/a2490c76-8ddf-4190-a31f-1de41d9142e0-kube-api-access-2tssv\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.152337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.152194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-serving-certs-ca-bundle\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.152337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.152242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.153594 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.153236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.153594 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.153441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-metrics-client-ca\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.154329 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.154278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2490c76-8ddf-4190-a31f-1de41d9142e0-serving-certs-ca-bundle\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.155262 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.155195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.155819 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.155794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-federate-client-tls\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.156284 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.156263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-secret-telemeter-client\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.157425 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.157400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2490c76-8ddf-4190-a31f-1de41d9142e0-telemeter-client-tls\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.163821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.163779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tssv\" (UniqueName: \"kubernetes.io/projected/a2490c76-8ddf-4190-a31f-1de41d9142e0-kube-api-access-2tssv\") pod \"telemeter-client-b6f7bd958-4l4hb\" (UID: \"a2490c76-8ddf-4190-a31f-1de41d9142e0\") " pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.171409 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.171342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"]
Sep 16 13:10:04.175696 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:10:04.175664    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8894ffc_e25f_45b0_b5ad_d078b2f069cb.slice/crio-c96019dd5a9bd87e42b85a6b309c83239f8ed7e7187efd6bf2df0abd3f59e20b WatchSource:0}: Error finding container c96019dd5a9bd87e42b85a6b309c83239f8ed7e7187efd6bf2df0abd3f59e20b: Status 404 returned error can't find the container with id c96019dd5a9bd87e42b85a6b309c83239f8ed7e7187efd6bf2df0abd3f59e20b
Sep 16 13:10:04.268183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.267902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"
Sep 16 13:10:04.465950 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.465911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c856fc477-mws7l" event={"ID":"149ded78-e698-4760-a6a8-4346af2a2db3","Type":"ContainerStarted","Data":"7386f707dc6bbc812da58aa6ad339f3360d94477ea4c429468f6d4de6ec0c261"}
Sep 16 13:10:04.469086 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.469051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndhbc" event={"ID":"9f39f477-f6c7-4b47-a65d-1810120be64c","Type":"ContainerStarted","Data":"e304ccfee2cf34a3db87444915d279a05c47029a057e4cd5d848a705e77ae160"}
Sep 16 13:10:04.469224 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.469114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndhbc" event={"ID":"9f39f477-f6c7-4b47-a65d-1810120be64c","Type":"ContainerStarted","Data":"9d2189b2fd72cfd0cf0d27560199d550dc4dc77d382fbba1a378e53c7b5acf5e"}
Sep 16 13:10:04.470654 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.470612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb" event={"ID":"f8894ffc-e25f-45b0-b5ad-d078b2f069cb","Type":"ContainerStarted","Data":"c96019dd5a9bd87e42b85a6b309c83239f8ed7e7187efd6bf2df0abd3f59e20b"}
Sep 16 13:10:04.498756 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.498712    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ndhbc" podStartSLOduration=3.585920517 podStartE2EDuration="6.498695918s" totalImagesPullingTime="2.814234802s" totalInitContainerRuntime="98.540599ms" isStatefulPod=true podCreationTimestamp="2026-09-16 13:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:04.496239118 +0000 UTC m=+75.918980697" watchObservedRunningTime="2026-09-16 13:10:04.498695918 +0000 UTC m=+75.921437496"
Sep 16 13:10:04.582424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:04.582334    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b6f7bd958-4l4hb"]
Sep 16 13:10:05.024932 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.024801    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:10:05.032842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.029456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.032842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.031994    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 13:10:05.039339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.038489    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 13:10:05.039339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.038665    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 13:10:05.039339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.038488    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 13:10:05.039339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.038909    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zmcc2\""
Sep 16 13:10:05.039339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.038959    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 13:10:05.039339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.038989    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 13:10:05.039339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.039242    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 13:10:05.039844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.039360    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 13:10:05.039844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.039452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 13:10:05.039844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.039513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-80rtp1nq7g0qa\""
Sep 16 13:10:05.039844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.039735    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 13:10:05.039844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.039735    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 13:10:05.039844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.039818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 13:10:05.046413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.046154    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 13:10:05.046413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.046183    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 13:10:05.061687 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.059701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.167926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.167972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjp5\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-kube-api-access-lgjp5\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.171601 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.172547 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.172547 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.172547 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.172547 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.168665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.269991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjp5\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-kube-api-access-lgjp5\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.270059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.270876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.270081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.271814 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.270108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.271814 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.270154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.271814 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.270173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.271814 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.270803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.272028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.271885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.272392 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.272127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.272392 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.272309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.273669 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.273644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.276394 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.276238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.276553 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.276510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.277969 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.277923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.279143 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.279119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.282707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.282683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjp5\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-kube-api-access-lgjp5\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.282707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.282694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.283196 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.283175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.283654 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.283629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.284090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.284048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.284471 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.284430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.285095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.285057    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.285628 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.285606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.286544 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.286521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.286954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.286931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.287963 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.287941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.364940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.364649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:05.364940 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:10:05.364877    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2490c76_8ddf_4190_a31f_1de41d9142e0.slice/crio-497714b3fa6251edd8ba39dee024a0a88dc66de2967f0e67383aa8fdf314daa2 WatchSource:0}: Error finding container 497714b3fa6251edd8ba39dee024a0a88dc66de2967f0e67383aa8fdf314daa2: Status 404 returned error can't find the container with id 497714b3fa6251edd8ba39dee024a0a88dc66de2967f0e67383aa8fdf314daa2
Sep 16 13:10:05.477607 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:05.477563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb" event={"ID":"a2490c76-8ddf-4190-a31f-1de41d9142e0","Type":"ContainerStarted","Data":"497714b3fa6251edd8ba39dee024a0a88dc66de2967f0e67383aa8fdf314daa2"}
Sep 16 13:10:06.230268 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.227283    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:10:06.489015 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.488979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerStarted","Data":"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd"}
Sep 16 13:10:06.489178 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.489067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerStarted","Data":"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc"}
Sep 16 13:10:06.489178 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.489105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerStarted","Data":"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702"}
Sep 16 13:10:06.489178 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.489143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerStarted","Data":"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f"}
Sep 16 13:10:06.491259 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.491129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerStarted","Data":"7b2e156572101bd3b78ba29e6a336c39bdd144d457ea502eb6a770eeb39f7f00"}
Sep 16 13:10:06.494894 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.494869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6" event={"ID":"52f757c9-aa31-4d10-9fcf-fe2b09a2103c","Type":"ContainerStarted","Data":"841a6f6a55bddaac8611201669671f473defc23bfb20c4ff118ede7c29729910"}
Sep 16 13:10:06.495021 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.494935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6" event={"ID":"52f757c9-aa31-4d10-9fcf-fe2b09a2103c","Type":"ContainerStarted","Data":"a1645f5a007ad912e9957f0555f81f2baf91b3e3d52e56597bd886dd4e8bb07c"}
Sep 16 13:10:06.495021 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:06.494973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6" event={"ID":"52f757c9-aa31-4d10-9fcf-fe2b09a2103c","Type":"ContainerStarted","Data":"67263a2d07871129727c92e55f79c68538e4917dba9e10a6d018bf297b61535e"}
Sep 16 13:10:08.510833 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.510322    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6" event={"ID":"52f757c9-aa31-4d10-9fcf-fe2b09a2103c","Type":"ContainerStarted","Data":"4b87ba065a6205a83712ca63df017b732e1ede0e3a9b78200c374e5f46cd771c"}
Sep 16 13:10:08.510833 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.510451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6" event={"ID":"52f757c9-aa31-4d10-9fcf-fe2b09a2103c","Type":"ContainerStarted","Data":"145b581e92fac156c947685d81a235fa7d0316b47067b06d64c073607923fc6c"}
Sep 16 13:10:08.510833 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.510493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6" event={"ID":"52f757c9-aa31-4d10-9fcf-fe2b09a2103c","Type":"ContainerStarted","Data":"6c15277b8c7f1c5221e7886db99d83f9df09c3d218973548f36c73a9871bf597"}
Sep 16 13:10:08.510833 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.510564    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:08.512562 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.512534    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb" event={"ID":"f8894ffc-e25f-45b0-b5ad-d078b2f069cb","Type":"ContainerStarted","Data":"82e757580db0d99973fce9ddf2174c0b69fc8ceb49611bcb9f15354cfab722e1"}
Sep 16 13:10:08.512737 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.512708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:08.517826 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.517707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerStarted","Data":"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113"}
Sep 16 13:10:08.517826 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.517779    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerStarted","Data":"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c"}
Sep 16 13:10:08.520205 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.520161    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb"
Sep 16 13:10:08.520615 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.520591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c856fc477-mws7l" event={"ID":"149ded78-e698-4760-a6a8-4346af2a2db3","Type":"ContainerStarted","Data":"3269273e8aa94bf22a49dc774143c28a1c6c406f0ba8f40b1c45c3e367a3972a"}
Sep 16 13:10:08.523243 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.523221    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb" event={"ID":"a2490c76-8ddf-4190-a31f-1de41d9142e0","Type":"ContainerStarted","Data":"b753cd76c027749eb174ddbfe2e034ba6f0c690794942c305cd913dacd38c37f"}
Sep 16 13:10:08.523808 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.523779    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb" event={"ID":"a2490c76-8ddf-4190-a31f-1de41d9142e0","Type":"ContainerStarted","Data":"b1ecde7880d15cf2317d8b313a91d1cc952c242d14ea11e0ef5b5bd65854eadf"}
Sep 16 13:10:08.523978 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.523955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb" event={"ID":"a2490c76-8ddf-4190-a31f-1de41d9142e0","Type":"ContainerStarted","Data":"27467727e095c30e504f22337b54f2b02b4186c0aacf8caf634efc4192b29f2c"}
Sep 16 13:10:08.525244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.525212    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6" exitCode=0
Sep 16 13:10:08.525332 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.525301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerDied","Data":"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"}
Sep 16 13:10:08.546152 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.546090    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6" podStartSLOduration=2.6617488419999997 podStartE2EDuration="7.546067863s" totalImagesPullingTime="4.884319021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:08.544525966 +0000 UTC m=+79.967267582" watchObservedRunningTime="2026-09-16 13:10:08.546067863 +0000 UTC m=+79.968809443"
Sep 16 13:10:08.593250 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.593198    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c856fc477-mws7l" podStartSLOduration=1.378740973 podStartE2EDuration="5.593175786s" totalImagesPullingTime="4.214434813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:08.591914783 +0000 UTC m=+80.014656378" watchObservedRunningTime="2026-09-16 13:10:08.593175786 +0000 UTC m=+80.015917356"
Sep 16 13:10:08.753098 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.753028    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.26786112 podStartE2EDuration="9.753003168s" totalImagesPullingTime="4.463944694s" totalInitContainerRuntime="21.197354ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:08.751702501 +0000 UTC m=+80.174444114" watchObservedRunningTime="2026-09-16 13:10:08.753003168 +0000 UTC m=+80.175744749"
Sep 16 13:10:08.789109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.788978    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b6f7bd958-4l4hb" podStartSLOduration=3.160001349 podStartE2EDuration="5.788955449s" totalImagesPullingTime="2.6289541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:08.787925429 +0000 UTC m=+80.210667009" watchObservedRunningTime="2026-09-16 13:10:08.788955449 +0000 UTC m=+80.211697028"
Sep 16 13:10:08.827027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:08.826969    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-84mgb" podStartSLOduration=2.010325612 podStartE2EDuration="5.826948762s" totalImagesPullingTime="3.81662315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:08.826029606 +0000 UTC m=+80.248771186" watchObservedRunningTime="2026-09-16 13:10:08.826948762 +0000 UTC m=+80.249690342"
Sep 16 13:10:14.539668 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:14.539538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5cf57bb6f8-f97v6"
Sep 16 13:10:19.584800 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:19.584764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerStarted","Data":"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"}
Sep 16 13:10:20.596170 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.596128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerStarted","Data":"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"}
Sep 16 13:10:20.596750 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.596231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerStarted","Data":"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"}
Sep 16 13:10:20.596750 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.596280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerStarted","Data":"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"}
Sep 16 13:10:20.596750 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.596317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerStarted","Data":"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"}
Sep 16 13:10:20.596750 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.596355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerStarted","Data":"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"}
Sep 16 13:10:20.598084 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.598050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-c4vg9" event={"ID":"78bf4632-a381-4760-bec8-3b52200112ac","Type":"ContainerStarted","Data":"30bf080a5335db7d693f3ed2e4f7afe84f518c3f4c05b9322604809610f82eb4"}
Sep 16 13:10:20.598271 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.598240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-c4vg9"
Sep 16 13:10:20.619903 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.619869    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-c4vg9"
Sep 16 13:10:20.638011 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.637942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.656577735 podStartE2EDuration="16.637914427s" totalImagesPullingTime="10.950225316s" totalInitContainerRuntime="31.111376ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:20.637210891 +0000 UTC m=+92.059952460" watchObservedRunningTime="2026-09-16 13:10:20.637914427 +0000 UTC m=+92.060656008"
Sep 16 13:10:20.660104 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:20.660044    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-c4vg9" podStartSLOduration=2.16735696 podStartE2EDuration="21.660024206s" totalImagesPullingTime="19.492667246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:20.658091267 +0000 UTC m=+92.080832846" watchObservedRunningTime="2026-09-16 13:10:20.660024206 +0000 UTC m=+92.082765785"
Sep 16 13:10:23.476568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:23.476526    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:23.477037 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:23.476605    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:25.365666 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:25.365626    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:10:31.442453 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:31.442410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vgrv9"
Sep 16 13:10:43.482242 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:43.482211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:10:43.486059 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:10:43.486030    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c856fc477-mws7l"
Sep 16 13:11:05.365574 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:05.365528    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:05.411930 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:05.411901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:05.812027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:05.812002    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:19.015044 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.015005    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:11:19.015602 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.015313    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="alertmanager" containerID="cri-o://f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f" gracePeriod=120
Sep 16 13:11:19.015602 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.015389    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy-metric" containerID="cri-o://4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c" gracePeriod=120
Sep 16 13:11:19.015602 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.015435    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="prom-label-proxy" containerID="cri-o://bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113" gracePeriod=120
Sep 16 13:11:19.015602 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.015400    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy-web" containerID="cri-o://03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc" gracePeriod=120
Sep 16 13:11:19.015602 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.015451    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy" containerID="cri-o://d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd" gracePeriod=120
Sep 16 13:11:19.015602 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.015480    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="config-reloader" containerID="cri-o://614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702" gracePeriod=120
Sep 16 13:11:19.854809 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.854774    2577 generic.go:356] "Generic (PLEG): container finished" podID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerID="bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113" exitCode=0
Sep 16 13:11:19.854809 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.854800    2577 generic.go:356] "Generic (PLEG): container finished" podID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerID="d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd" exitCode=0
Sep 16 13:11:19.854809 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.854807    2577 generic.go:356] "Generic (PLEG): container finished" podID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerID="614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702" exitCode=0
Sep 16 13:11:19.854809 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.854815    2577 generic.go:356] "Generic (PLEG): container finished" podID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerID="f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f" exitCode=0
Sep 16 13:11:19.855090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.854842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerDied","Data":"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113"}
Sep 16 13:11:19.855090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.854918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerDied","Data":"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd"}
Sep 16 13:11:19.855090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.854951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerDied","Data":"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702"}
Sep 16 13:11:19.855090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:19.854979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerDied","Data":"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f"}
Sep 16 13:11:20.266086 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.266059    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.359039 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-main-db\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359063    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-metric\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-tls-assets\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359116    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-web\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359139    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-cluster-tls-config\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-main-tls\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-metrics-client-ca\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359241    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-out\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-trusted-ca-bundle\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359316    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-web-config\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359345    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359344    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:20.359531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsd9p\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-kube-api-access-gsd9p\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.359531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359428    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-volume\") pod \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\" (UID: \"68bdd8f3-2aea-448f-8214-e7ed9e38db29\") "
Sep 16 13:11:20.361096 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359661    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-main-db\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.361096 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.359688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:20.361096 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.360007    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:20.362068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.362024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:20.362461 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.362433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:20.362584 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.362561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:11:20.362869 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.362755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:20.363051 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.363027    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:20.363258 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.363239    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-volume" (OuterVolumeSpecName: "config-volume") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:20.364007 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.363988    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-out" (OuterVolumeSpecName: "config-out") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:20.364566 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.364543    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-kube-api-access-gsd9p" (OuterVolumeSpecName: "kube-api-access-gsd9p") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "kube-api-access-gsd9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:11:20.367282 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.367256    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:20.373986 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.373929    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-web-config" (OuterVolumeSpecName: "web-config") pod "68bdd8f3-2aea-448f-8214-e7ed9e38db29" (UID: "68bdd8f3-2aea-448f-8214-e7ed9e38db29"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:20.460102 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460061    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460102 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460094    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460102 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460108    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460123    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-cluster-tls-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460136    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-main-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460148    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-metrics-client-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460159    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460170    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bdd8f3-2aea-448f-8214-e7ed9e38db29-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460180    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460192    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460203    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsd9p\" (UniqueName: \"kubernetes.io/projected/68bdd8f3-2aea-448f-8214-e7ed9e38db29-kube-api-access-gsd9p\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.460346 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.460214    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68bdd8f3-2aea-448f-8214-e7ed9e38db29-config-volume\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:20.862449 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.862409    2577 generic.go:356] "Generic (PLEG): container finished" podID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerID="4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c" exitCode=0
Sep 16 13:11:20.862449 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.862442    2577 generic.go:356] "Generic (PLEG): container finished" podID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerID="03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc" exitCode=0
Sep 16 13:11:20.862673 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.862482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerDied","Data":"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c"}
Sep 16 13:11:20.862673 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.862517    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.862673 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.862566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerDied","Data":"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc"}
Sep 16 13:11:20.862673 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.862611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68bdd8f3-2aea-448f-8214-e7ed9e38db29","Type":"ContainerDied","Data":"ce1f99c08dd9299a3c634b136ff8c6627a7dfba950f2530780cd8a6e6fd2079b"}
Sep 16 13:11:20.862673 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.862655    2577 scope.go:117] "RemoveContainer" containerID="bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113"
Sep 16 13:11:20.872672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.872652    2577 scope.go:117] "RemoveContainer" containerID="4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c"
Sep 16 13:11:20.880256 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.880240    2577 scope.go:117] "RemoveContainer" containerID="d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd"
Sep 16 13:11:20.887500 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.887482    2577 scope.go:117] "RemoveContainer" containerID="03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc"
Sep 16 13:11:20.888004 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.887984    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:11:20.892470 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.892450    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:11:20.895147 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.895129    2577 scope.go:117] "RemoveContainer" containerID="614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702"
Sep 16 13:11:20.901910 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.901895    2577 scope.go:117] "RemoveContainer" containerID="f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f"
Sep 16 13:11:20.909240 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.909219    2577 scope.go:117] "RemoveContainer" containerID="c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac"
Sep 16 13:11:20.916287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.916142    2577 scope.go:117] "RemoveContainer" containerID="bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113"
Sep 16 13:11:20.916882 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:20.916489    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113\": container with ID starting with bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113 not found: ID does not exist" containerID="bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113"
Sep 16 13:11:20.916882 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.916529    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113"} err="failed to get container status \"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113\": rpc error: code = NotFound desc = could not find container \"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113\": container with ID starting with bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113 not found: ID does not exist"
Sep 16 13:11:20.916882 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.916575    2577 scope.go:117] "RemoveContainer" containerID="4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c"
Sep 16 13:11:20.917099 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:20.916975    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c\": container with ID starting with 4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c not found: ID does not exist" containerID="4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c"
Sep 16 13:11:20.917099 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.917008    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c"} err="failed to get container status \"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c\": rpc error: code = NotFound desc = could not find container \"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c\": container with ID starting with 4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c not found: ID does not exist"
Sep 16 13:11:20.917099 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.917031    2577 scope.go:117] "RemoveContainer" containerID="d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:20.917635    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd\": container with ID starting with d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd not found: ID does not exist" containerID="d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.917722    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd"} err="failed to get container status \"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd\": rpc error: code = NotFound desc = could not find container \"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd\": container with ID starting with d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd not found: ID does not exist"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.917746    2577 scope.go:117] "RemoveContainer" containerID="03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919143    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919811    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy-web"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919828    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy-web"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919841    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="prom-label-proxy"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919869    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="prom-label-proxy"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919896    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy-metric"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919904    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy-metric"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919922    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="config-reloader"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919951    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="config-reloader"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919973    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="init-config-reloader"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.919983    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="init-config-reloader"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920016    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="alertmanager"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920025    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="alertmanager"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920042    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920051    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920185    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="alertmanager"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920197    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="config-reloader"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920208    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy-web"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920226    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy-metric"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920237    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="kube-rbac-proxy"
Sep 16 13:11:20.921692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.920252    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" containerName="prom-label-proxy"
Sep 16 13:11:20.923204 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:20.923082    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc\": container with ID starting with 03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc not found: ID does not exist" containerID="03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc"
Sep 16 13:11:20.923204 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.923149    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc"} err="failed to get container status \"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc\": rpc error: code = NotFound desc = could not find container \"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc\": container with ID starting with 03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc not found: ID does not exist"
Sep 16 13:11:20.923204 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.923172    2577 scope.go:117] "RemoveContainer" containerID="614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702"
Sep 16 13:11:20.923730 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:20.923523    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702\": container with ID starting with 614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702 not found: ID does not exist" containerID="614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702"
Sep 16 13:11:20.923730 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.923565    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702"} err="failed to get container status \"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702\": rpc error: code = NotFound desc = could not find container \"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702\": container with ID starting with 614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702 not found: ID does not exist"
Sep 16 13:11:20.923730 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.923588    2577 scope.go:117] "RemoveContainer" containerID="f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f"
Sep 16 13:11:20.925326 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:20.923997    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f\": container with ID starting with f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f not found: ID does not exist" containerID="f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f"
Sep 16 13:11:20.925326 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.924041    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f"} err="failed to get container status \"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f\": rpc error: code = NotFound desc = could not find container \"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f\": container with ID starting with f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f not found: ID does not exist"
Sep 16 13:11:20.925326 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.924063    2577 scope.go:117] "RemoveContainer" containerID="c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac"
Sep 16 13:11:20.925531 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:20.925475    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac\": container with ID starting with c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac not found: ID does not exist" containerID="c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac"
Sep 16 13:11:20.925531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.925500    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac"} err="failed to get container status \"c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac\": rpc error: code = NotFound desc = could not find container \"c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac\": container with ID starting with c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac not found: ID does not exist"
Sep 16 13:11:20.925531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.925515    2577 scope.go:117] "RemoveContainer" containerID="bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113"
Sep 16 13:11:20.925818 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.925793    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113"} err="failed to get container status \"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113\": rpc error: code = NotFound desc = could not find container \"bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113\": container with ID starting with bdf2dafa5f5a3e54ee7419cb192f3c3436915e2d1cf51a226e6326dfa07e3113 not found: ID does not exist"
Sep 16 13:11:20.925954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.925934    2577 scope.go:117] "RemoveContainer" containerID="4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c"
Sep 16 13:11:20.926278 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.926251    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c"} err="failed to get container status \"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c\": rpc error: code = NotFound desc = could not find container \"4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c\": container with ID starting with 4dc4ed3bdb1632b31b15f41413c5e1483be00cb8f76c79e3ece329a7b7a9e49c not found: ID does not exist"
Sep 16 13:11:20.926344 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.926277    2577 scope.go:117] "RemoveContainer" containerID="d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd"
Sep 16 13:11:20.926557 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.926532    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd"} err="failed to get container status \"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd\": rpc error: code = NotFound desc = could not find container \"d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd\": container with ID starting with d9ad3e9657f65c5f257dd532fa912c91425f7248307d75d982256393249df3cd not found: ID does not exist"
Sep 16 13:11:20.926606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.926567    2577 scope.go:117] "RemoveContainer" containerID="03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc"
Sep 16 13:11:20.926934 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.926900    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc"} err="failed to get container status \"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc\": rpc error: code = NotFound desc = could not find container \"03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc\": container with ID starting with 03d404b81b857668a9b369cb6cbf66524c4d0b1c5dfe2db7e06e406174e032dc not found: ID does not exist"
Sep 16 13:11:20.926934 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.926936    2577 scope.go:117] "RemoveContainer" containerID="614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702"
Sep 16 13:11:20.927206 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.927185    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702"} err="failed to get container status \"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702\": rpc error: code = NotFound desc = could not find container \"614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702\": container with ID starting with 614800e013b882b743f05475903f07aa6f8bf5b26751e49c8ffe9b3a4c7ba702 not found: ID does not exist"
Sep 16 13:11:20.927206 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.927204    2577 scope.go:117] "RemoveContainer" containerID="f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f"
Sep 16 13:11:20.927492 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.927479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.927564 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.927478    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f"} err="failed to get container status \"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f\": rpc error: code = NotFound desc = could not find container \"f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f\": container with ID starting with f6bf24c7d3b186084541cb4fd714f7cea785e1bfa6838efad6a181cfe9e0fd0f not found: ID does not exist"
Sep 16 13:11:20.927604 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.927565    2577 scope.go:117] "RemoveContainer" containerID="c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac"
Sep 16 13:11:20.927844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.927811    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac"} err="failed to get container status \"c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac\": rpc error: code = NotFound desc = could not find container \"c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac\": container with ID starting with c47f4308bc17cfc827977ba1c4f699c61c5dd997f50746c201ffba00d45533ac not found: ID does not exist"
Sep 16 13:11:20.930019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.930003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 13:11:20.930092 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.930050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 13:11:20.932193 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.932174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 13:11:20.932287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.932214    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 13:11:20.932287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.932234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 13:11:20.932287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.932237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 13:11:20.932287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.932237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 13:11:20.932287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.932285    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r9j2b\""
Sep 16 13:11:20.932509 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.932246    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 13:11:20.937482 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.937465    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 13:11:20.941683 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.941665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:11:20.963647 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f630ad5-4918-4957-a9ab-81fcd607aed0-config-out\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.963776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f630ad5-4918-4957-a9ab-81fcd607aed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.963776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.963776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.963776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f630ad5-4918-4957-a9ab-81fcd607aed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.963981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f630ad5-4918-4957-a9ab-81fcd607aed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.963981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.963981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f630ad5-4918-4957-a9ab-81fcd607aed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.963981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t75z\" (UniqueName: \"kubernetes.io/projected/3f630ad5-4918-4957-a9ab-81fcd607aed0-kube-api-access-5t75z\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.964164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.963978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.964164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.964020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.964164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.964052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:20.964164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:20.964079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-web-config\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.064639 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.064604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.064639 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.064646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.064928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.064667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.064928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.064733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-web-config\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.064928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.064776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f630ad5-4918-4957-a9ab-81fcd607aed0-config-out\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.064928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.064800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f630ad5-4918-4957-a9ab-81fcd607aed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.064928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.064829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.064928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.064887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.065229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.065064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f630ad5-4918-4957-a9ab-81fcd607aed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.065229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.065100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f630ad5-4918-4957-a9ab-81fcd607aed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.065229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.065149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.065229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.065185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f630ad5-4918-4957-a9ab-81fcd607aed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.065229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.065221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t75z\" (UniqueName: \"kubernetes.io/projected/3f630ad5-4918-4957-a9ab-81fcd607aed0-kube-api-access-5t75z\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.065468 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.065335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f630ad5-4918-4957-a9ab-81fcd607aed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.066128 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.066100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f630ad5-4918-4957-a9ab-81fcd607aed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.066560 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.066531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f630ad5-4918-4957-a9ab-81fcd607aed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.067912 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.067879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.068050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.068027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.068147 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.068125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.068207 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.068172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f630ad5-4918-4957-a9ab-81fcd607aed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.068436 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.068413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.068552 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.068524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.068961 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.068941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.069083 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.069069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f630ad5-4918-4957-a9ab-81fcd607aed0-config-out\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.069379 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.069364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f630ad5-4918-4957-a9ab-81fcd607aed0-web-config\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.073045 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.073021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t75z\" (UniqueName: \"kubernetes.io/projected/3f630ad5-4918-4957-a9ab-81fcd607aed0-kube-api-access-5t75z\") pod \"alertmanager-main-0\" (UID: \"3f630ad5-4918-4957-a9ab-81fcd607aed0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.108328 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.108300    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68bdd8f3-2aea-448f-8214-e7ed9e38db29" path="/var/lib/kubelet/pods/68bdd8f3-2aea-448f-8214-e7ed9e38db29/volumes"
Sep 16 13:11:21.237803 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.237759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 13:11:21.522195 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.522172    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 13:11:21.524585 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:11:21.524556    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f630ad5_4918_4957_a9ab_81fcd607aed0.slice/crio-41e7d572ae658f61dc4c852f330d40c2c05da1832b5ac51dfdd41384666db4b4 WatchSource:0}: Error finding container 41e7d572ae658f61dc4c852f330d40c2c05da1832b5ac51dfdd41384666db4b4: Status 404 returned error can't find the container with id 41e7d572ae658f61dc4c852f330d40c2c05da1832b5ac51dfdd41384666db4b4
Sep 16 13:11:21.869117 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.869031    2577 generic.go:356] "Generic (PLEG): container finished" podID="3f630ad5-4918-4957-a9ab-81fcd607aed0" containerID="e4bab8ba3230eaea62fbfef6ed79fb7b19b4f976ef29f3f0279a982f1dc3614d" exitCode=0
Sep 16 13:11:21.869117 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.869111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f630ad5-4918-4957-a9ab-81fcd607aed0","Type":"ContainerDied","Data":"e4bab8ba3230eaea62fbfef6ed79fb7b19b4f976ef29f3f0279a982f1dc3614d"}
Sep 16 13:11:21.869301 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:21.869163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f630ad5-4918-4957-a9ab-81fcd607aed0","Type":"ContainerStarted","Data":"41e7d572ae658f61dc4c852f330d40c2c05da1832b5ac51dfdd41384666db4b4"}
Sep 16 13:11:22.876956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:22.876920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f630ad5-4918-4957-a9ab-81fcd607aed0","Type":"ContainerStarted","Data":"0f791d0e9919da952e474a574901ae3164d4a7371b6f7da9f8d5063c695118dd"}
Sep 16 13:11:22.877375 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:22.876984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f630ad5-4918-4957-a9ab-81fcd607aed0","Type":"ContainerStarted","Data":"0f232264c6e14c69fc351770b7f05a6bf22d7a0613f613bbca24583b1ab462bc"}
Sep 16 13:11:22.877375 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:22.877008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f630ad5-4918-4957-a9ab-81fcd607aed0","Type":"ContainerStarted","Data":"4e98db43ce36ea1ec0e709c3bfd80a7bb133f4b76b6280a18431a6b925d83896"}
Sep 16 13:11:22.877375 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:22.877033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f630ad5-4918-4957-a9ab-81fcd607aed0","Type":"ContainerStarted","Data":"24cea9a96059ea0a49e0f7838642fbf5211e227078de5cc0d6ada61b97f0f0f7"}
Sep 16 13:11:22.877375 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:22.877059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f630ad5-4918-4957-a9ab-81fcd607aed0","Type":"ContainerStarted","Data":"7e69842bc1dda388936b9c053ceb0e479bf06d2f13ec314c4b29e662d138db90"}
Sep 16 13:11:22.877375 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:22.877082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f630ad5-4918-4957-a9ab-81fcd607aed0","Type":"ContainerStarted","Data":"e2fccac70e62629627b8ea227beed26bcdd9061a4605614ade9ed861a251214c"}
Sep 16 13:11:22.905247 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:22.905185    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.888759834 podStartE2EDuration="2.905166943s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.407109ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:11:22.903032613 +0000 UTC m=+154.325774218" watchObservedRunningTime="2026-09-16 13:11:22.905166943 +0000 UTC m=+154.327908522"
Sep 16 13:11:23.263350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.263312    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:11:23.263939 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.263903    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="prometheus" containerID="cri-o://97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1" gracePeriod=600
Sep 16 13:11:23.264106 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.264078    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy-web" containerID="cri-o://9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e" gracePeriod=600
Sep 16 13:11:23.264212 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.263986    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="thanos-sidecar" containerID="cri-o://033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b" gracePeriod=600
Sep 16 13:11:23.264274 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.263986    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy" containerID="cri-o://5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7" gracePeriod=600
Sep 16 13:11:23.264274 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.264015    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="config-reloader" containerID="cri-o://b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9" gracePeriod=600
Sep 16 13:11:23.264385 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.264147    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b" gracePeriod=600
Sep 16 13:11:23.522576 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.522515    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:23.591250 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-tls-assets\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591250 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-tls\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-serving-certs-ca-bundle\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgjp5\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-kube-api-access-lgjp5\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-0\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-trusted-ca-bundle\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-metrics-client-ca\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-db\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-web-config\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-2\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-thanos-prometheus-http-client-file\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-kube-rbac-proxy\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591603    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-kubelet-serving-ca-bundle\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-grpc-tls\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591679    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-metrics-client-certs\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591706    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-1\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.591902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config-out\") pod \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\" (UID: \"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed\") "
Sep 16 13:11:23.592469 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.591986    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:23.592469 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.592176    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.592469 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.592195    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.592815 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.592784    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:23.593003 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.592973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:23.593361 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.593336    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:23.594270 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.594240    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:23.594356 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.594292    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:23.594585 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.594558    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:11:23.594680 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.594640    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-kube-api-access-lgjp5" (OuterVolumeSpecName: "kube-api-access-lgjp5") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "kube-api-access-lgjp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:11:23.594933 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.594887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config-out" (OuterVolumeSpecName: "config-out") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:23.595414 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.595381    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:11:23.595687 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.595628    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.595687 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.595655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.596701 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.596679    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.596796 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.596723    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.597322 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.597295    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.597322 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.597313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config" (OuterVolumeSpecName: "config") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.597467 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.597434    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.597662 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.597640    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.605803 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.605783    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-web-config" (OuterVolumeSpecName: "web-config") pod "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" (UID: "6b7445c4-3539-4e5d-81e7-aeef6e19e1ed"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:23.692876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692825    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-db\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.692876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692865    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.692876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692877    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.692876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692886    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-kube-rbac-proxy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692896    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692904    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-grpc-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692913    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-metrics-client-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692921    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692929    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692938    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692949    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692956    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692965    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692972    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692980    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692988    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjp5\" (UniqueName: \"kubernetes.io/projected/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-kube-api-access-lgjp5\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.692996    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.693121 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.693020    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed-configmap-metrics-client-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:11:23.884296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884206    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b" exitCode=0
Sep 16 13:11:23.884296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884230    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7" exitCode=0
Sep 16 13:11:23.884296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884236    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e" exitCode=0
Sep 16 13:11:23.884296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884242    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b" exitCode=0
Sep 16 13:11:23.884296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884247    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9" exitCode=0
Sep 16 13:11:23.884296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884252    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1" exitCode=0
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerDied","Data":"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"}
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884317    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerDied","Data":"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"}
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerDied","Data":"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"}
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884406    2577 scope.go:117] "RemoveContainer" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerDied","Data":"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"}
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerDied","Data":"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"}
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerDied","Data":"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"}
Sep 16 13:11:23.885018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.884495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7445c4-3539-4e5d-81e7-aeef6e19e1ed","Type":"ContainerDied","Data":"7b2e156572101bd3b78ba29e6a336c39bdd144d457ea502eb6a770eeb39f7f00"}
Sep 16 13:11:23.893165 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.893146    2577 scope.go:117] "RemoveContainer" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"
Sep 16 13:11:23.900446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.900428    2577 scope.go:117] "RemoveContainer" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"
Sep 16 13:11:23.907830 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.907785    2577 scope.go:117] "RemoveContainer" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"
Sep 16 13:11:23.909022 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.908997    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:11:23.912513 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.912491    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:11:23.916064 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.916044    2577 scope.go:117] "RemoveContainer" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"
Sep 16 13:11:23.922919 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.922846    2577 scope.go:117] "RemoveContainer" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"
Sep 16 13:11:23.930085 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.930065    2577 scope.go:117] "RemoveContainer" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"
Sep 16 13:11:23.937031 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.937010    2577 scope.go:117] "RemoveContainer" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"
Sep 16 13:11:23.937301 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:23.937283    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": container with ID starting with e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b not found: ID does not exist" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"
Sep 16 13:11:23.937355 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.937310    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"} err="failed to get container status \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": rpc error: code = NotFound desc = could not find container \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": container with ID starting with e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b not found: ID does not exist"
Sep 16 13:11:23.937355 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.937329    2577 scope.go:117] "RemoveContainer" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"
Sep 16 13:11:23.937575 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:23.937556    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": container with ID starting with 5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7 not found: ID does not exist" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"
Sep 16 13:11:23.937654 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.937579    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"} err="failed to get container status \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": rpc error: code = NotFound desc = could not find container \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": container with ID starting with 5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7 not found: ID does not exist"
Sep 16 13:11:23.937654 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.937605    2577 scope.go:117] "RemoveContainer" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"
Sep 16 13:11:23.937877 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:23.937837    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": container with ID starting with 9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e not found: ID does not exist" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"
Sep 16 13:11:23.937944 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.937881    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"} err="failed to get container status \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": rpc error: code = NotFound desc = could not find container \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": container with ID starting with 9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e not found: ID does not exist"
Sep 16 13:11:23.937944 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.937902    2577 scope.go:117] "RemoveContainer" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"
Sep 16 13:11:23.938156 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:23.938139    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": container with ID starting with 033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b not found: ID does not exist" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"
Sep 16 13:11:23.938201 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.938161    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"} err="failed to get container status \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": rpc error: code = NotFound desc = could not find container \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": container with ID starting with 033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b not found: ID does not exist"
Sep 16 13:11:23.938201 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.938176    2577 scope.go:117] "RemoveContainer" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"
Sep 16 13:11:23.938431 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:23.938411    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": container with ID starting with b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9 not found: ID does not exist" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"
Sep 16 13:11:23.938540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.938435    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"} err="failed to get container status \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": rpc error: code = NotFound desc = could not find container \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": container with ID starting with b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9 not found: ID does not exist"
Sep 16 13:11:23.938540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.938450    2577 scope.go:117] "RemoveContainer" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"
Sep 16 13:11:23.938756 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:23.938739    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": container with ID starting with 97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1 not found: ID does not exist" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"
Sep 16 13:11:23.938816 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.938759    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"} err="failed to get container status \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": rpc error: code = NotFound desc = could not find container \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": container with ID starting with 97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1 not found: ID does not exist"
Sep 16 13:11:23.938816 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.938778    2577 scope.go:117] "RemoveContainer" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"
Sep 16 13:11:23.939106 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:11:23.939084    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": container with ID starting with c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6 not found: ID does not exist" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"
Sep 16 13:11:23.939170 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.939113    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"} err="failed to get container status \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": rpc error: code = NotFound desc = could not find container \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": container with ID starting with c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6 not found: ID does not exist"
Sep 16 13:11:23.939170 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.939132    2577 scope.go:117] "RemoveContainer" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"
Sep 16 13:11:23.939409 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.939391    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"} err="failed to get container status \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": rpc error: code = NotFound desc = could not find container \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": container with ID starting with e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b not found: ID does not exist"
Sep 16 13:11:23.939471 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.939411    2577 scope.go:117] "RemoveContainer" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"
Sep 16 13:11:23.939727 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.939708    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"} err="failed to get container status \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": rpc error: code = NotFound desc = could not find container \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": container with ID starting with 5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7 not found: ID does not exist"
Sep 16 13:11:23.939773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.939729    2577 scope.go:117] "RemoveContainer" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"
Sep 16 13:11:23.940050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.940030    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"} err="failed to get container status \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": rpc error: code = NotFound desc = could not find container \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": container with ID starting with 9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e not found: ID does not exist"
Sep 16 13:11:23.940050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.940052    2577 scope.go:117] "RemoveContainer" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"
Sep 16 13:11:23.940406 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.940379    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"} err="failed to get container status \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": rpc error: code = NotFound desc = could not find container \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": container with ID starting with 033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b not found: ID does not exist"
Sep 16 13:11:23.940406 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.940406    2577 scope.go:117] "RemoveContainer" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"
Sep 16 13:11:23.940747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.940681    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"} err="failed to get container status \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": rpc error: code = NotFound desc = could not find container \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": container with ID starting with b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9 not found: ID does not exist"
Sep 16 13:11:23.940747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.940708    2577 scope.go:117] "RemoveContainer" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"
Sep 16 13:11:23.941013 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.940989    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"} err="failed to get container status \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": rpc error: code = NotFound desc = could not find container \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": container with ID starting with 97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1 not found: ID does not exist"
Sep 16 13:11:23.941099 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.941016    2577 scope.go:117] "RemoveContainer" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"
Sep 16 13:11:23.941315 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.941297    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"} err="failed to get container status \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": rpc error: code = NotFound desc = could not find container \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": container with ID starting with c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6 not found: ID does not exist"
Sep 16 13:11:23.941376 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.941316    2577 scope.go:117] "RemoveContainer" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"
Sep 16 13:11:23.941611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.941591    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"} err="failed to get container status \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": rpc error: code = NotFound desc = could not find container \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": container with ID starting with e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b not found: ID does not exist"
Sep 16 13:11:23.941665 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.941612    2577 scope.go:117] "RemoveContainer" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"
Sep 16 13:11:23.941868 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.941833    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"} err="failed to get container status \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": rpc error: code = NotFound desc = could not find container \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": container with ID starting with 5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7 not found: ID does not exist"
Sep 16 13:11:23.941929 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.941870    2577 scope.go:117] "RemoveContainer" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"
Sep 16 13:11:23.942089 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942068    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:11:23.942164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942095    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"} err="failed to get container status \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": rpc error: code = NotFound desc = could not find container \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": container with ID starting with 9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e not found: ID does not exist"
Sep 16 13:11:23.942164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942112    2577 scope.go:117] "RemoveContainer" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"
Sep 16 13:11:23.942361 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942344    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"} err="failed to get container status \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": rpc error: code = NotFound desc = could not find container \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": container with ID starting with 033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b not found: ID does not exist"
Sep 16 13:11:23.942400 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942363    2577 scope.go:117] "RemoveContainer" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"
Sep 16 13:11:23.942447 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942411    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy-web"
Sep 16 13:11:23.942447 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942424    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy-web"
Sep 16 13:11:23.942447 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942441    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="prometheus"
Sep 16 13:11:23.942447 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942447    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="prometheus"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942454    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942460    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942478    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy-thanos"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942487    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy-thanos"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942496    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="thanos-sidecar"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942511    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="thanos-sidecar"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942522    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="init-config-reloader"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942527    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="init-config-reloader"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942536    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="config-reloader"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942542    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="config-reloader"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942600    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="prometheus"
Sep 16 13:11:23.942616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942612    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="config-reloader"
Sep 16 13:11:23.943052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942615    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"} err="failed to get container status \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": rpc error: code = NotFound desc = could not find container \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": container with ID starting with b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9 not found: ID does not exist"
Sep 16 13:11:23.943052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942634    2577 scope.go:117] "RemoveContainer" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"
Sep 16 13:11:23.943052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942623    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy-thanos"
Sep 16 13:11:23.943052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942679    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy-web"
Sep 16 13:11:23.943052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942689    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="thanos-sidecar"
Sep 16 13:11:23.943052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942705    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" containerName="kube-rbac-proxy"
Sep 16 13:11:23.943052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942892    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"} err="failed to get container status \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": rpc error: code = NotFound desc = could not find container \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": container with ID starting with 97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1 not found: ID does not exist"
Sep 16 13:11:23.943052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.942906    2577 scope.go:117] "RemoveContainer" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"
Sep 16 13:11:23.943350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.943135    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"} err="failed to get container status \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": rpc error: code = NotFound desc = could not find container \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": container with ID starting with c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6 not found: ID does not exist"
Sep 16 13:11:23.943350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.943154    2577 scope.go:117] "RemoveContainer" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"
Sep 16 13:11:23.943423 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.943380    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"} err="failed to get container status \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": rpc error: code = NotFound desc = could not find container \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": container with ID starting with e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b not found: ID does not exist"
Sep 16 13:11:23.943473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.943425    2577 scope.go:117] "RemoveContainer" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"
Sep 16 13:11:23.943743 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.943684    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"} err="failed to get container status \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": rpc error: code = NotFound desc = could not find container \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": container with ID starting with 5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7 not found: ID does not exist"
Sep 16 13:11:23.943743 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.943732    2577 scope.go:117] "RemoveContainer" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"
Sep 16 13:11:23.944114 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.944094    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"} err="failed to get container status \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": rpc error: code = NotFound desc = could not find container \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": container with ID starting with 9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e not found: ID does not exist"
Sep 16 13:11:23.944114 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.944114    2577 scope.go:117] "RemoveContainer" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"
Sep 16 13:11:23.944377 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.944357    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"} err="failed to get container status \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": rpc error: code = NotFound desc = could not find container \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": container with ID starting with 033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b not found: ID does not exist"
Sep 16 13:11:23.944462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.944378    2577 scope.go:117] "RemoveContainer" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"
Sep 16 13:11:23.944644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.944627    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"} err="failed to get container status \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": rpc error: code = NotFound desc = could not find container \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": container with ID starting with b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9 not found: ID does not exist"
Sep 16 13:11:23.944713 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.944645    2577 scope.go:117] "RemoveContainer" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"
Sep 16 13:11:23.944979 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.944952    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"} err="failed to get container status \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": rpc error: code = NotFound desc = could not find container \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": container with ID starting with 97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1 not found: ID does not exist"
Sep 16 13:11:23.944979 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.944978    2577 scope.go:117] "RemoveContainer" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.945560    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"} err="failed to get container status \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": rpc error: code = NotFound desc = could not find container \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": container with ID starting with c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6 not found: ID does not exist"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.945586    2577 scope.go:117] "RemoveContainer" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.945989    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"} err="failed to get container status \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": rpc error: code = NotFound desc = could not find container \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": container with ID starting with e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b not found: ID does not exist"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.946011    2577 scope.go:117] "RemoveContainer" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.946313    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"} err="failed to get container status \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": rpc error: code = NotFound desc = could not find container \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": container with ID starting with 5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7 not found: ID does not exist"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.946334    2577 scope.go:117] "RemoveContainer" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.946606    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"} err="failed to get container status \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": rpc error: code = NotFound desc = could not find container \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": container with ID starting with 9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e not found: ID does not exist"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.946643    2577 scope.go:117] "RemoveContainer" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.946971    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"} err="failed to get container status \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": rpc error: code = NotFound desc = could not find container \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": container with ID starting with 033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b not found: ID does not exist"
Sep 16 13:11:23.947109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.946993    2577 scope.go:117] "RemoveContainer" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"
Sep 16 13:11:23.947606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.947255    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"} err="failed to get container status \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": rpc error: code = NotFound desc = could not find container \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": container with ID starting with b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9 not found: ID does not exist"
Sep 16 13:11:23.947606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.947274    2577 scope.go:117] "RemoveContainer" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"
Sep 16 13:11:23.947606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.947523    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"} err="failed to get container status \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": rpc error: code = NotFound desc = could not find container \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": container with ID starting with 97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1 not found: ID does not exist"
Sep 16 13:11:23.947606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.947547    2577 scope.go:117] "RemoveContainer" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"
Sep 16 13:11:23.947844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.947826    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"} err="failed to get container status \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": rpc error: code = NotFound desc = could not find container \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": container with ID starting with c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6 not found: ID does not exist"
Sep 16 13:11:23.947903 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.947845    2577 scope.go:117] "RemoveContainer" containerID="e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"
Sep 16 13:11:23.948131 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.948114    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b"} err="failed to get container status \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": rpc error: code = NotFound desc = could not find container \"e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b\": container with ID starting with e91eb8730b733e528907d1233e64be7ae14536b43d19f6c1fece0f4746d6432b not found: ID does not exist"
Sep 16 13:11:23.948179 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.948130    2577 scope.go:117] "RemoveContainer" containerID="5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"
Sep 16 13:11:23.948301 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.948284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:23.948377 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.948359    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7"} err="failed to get container status \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": rpc error: code = NotFound desc = could not find container \"5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7\": container with ID starting with 5232f2bd87fcbc1558110e6eb804c308e5c8a0db93e9a62dc6ac0595a59413a7 not found: ID does not exist"
Sep 16 13:11:23.948426 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.948378    2577 scope.go:117] "RemoveContainer" containerID="9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"
Sep 16 13:11:23.948669 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.948639    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e"} err="failed to get container status \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": rpc error: code = NotFound desc = could not find container \"9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e\": container with ID starting with 9cc517afebd5a964a3cf76d06d943ff39bb4a704ed74ecc626356ee2086ff12e not found: ID does not exist"
Sep 16 13:11:23.948763 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.948672    2577 scope.go:117] "RemoveContainer" containerID="033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"
Sep 16 13:11:23.949018 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.948986    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b"} err="failed to get container status \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": rpc error: code = NotFound desc = could not find container \"033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b\": container with ID starting with 033fd80d7dfcc4847dd3751612a40c4ef28c2a3d9d30dbe9aa3f0771a8113c3b not found: ID does not exist"
Sep 16 13:11:23.949098 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.949018    2577 scope.go:117] "RemoveContainer" containerID="b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"
Sep 16 13:11:23.949302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.949285    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9"} err="failed to get container status \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": rpc error: code = NotFound desc = could not find container \"b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9\": container with ID starting with b01e91cc52a95acc24582149ae458a8af2cfc9776e0627b7dc41b2e07c6f29c9 not found: ID does not exist"
Sep 16 13:11:23.949358 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.949302    2577 scope.go:117] "RemoveContainer" containerID="97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"
Sep 16 13:11:23.949523 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.949509    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1"} err="failed to get container status \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": rpc error: code = NotFound desc = could not find container \"97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1\": container with ID starting with 97af5d695cbbe9e9eba4486e936931073f8b88ec2921506af73bf945cd6046b1 not found: ID does not exist"
Sep 16 13:11:23.949580 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.949525    2577 scope.go:117] "RemoveContainer" containerID="c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"
Sep 16 13:11:23.949830 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.949812    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6"} err="failed to get container status \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": rpc error: code = NotFound desc = could not find container \"c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6\": container with ID starting with c1950c84d451ec0bc75829f74f2eff6098fc2e6fb1d82bd10c709a8cac2a7cd6 not found: ID does not exist"
Sep 16 13:11:23.950595 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.950574    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 13:11:23.950595 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.950574    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 13:11:23.950737 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.950617    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-80rtp1nq7g0qa\""
Sep 16 13:11:23.950949 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.950927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 13:11:23.951086 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951047    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 13:11:23.951235 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951208    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 13:11:23.951235 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951218    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 13:11:23.951442 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951259    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 13:11:23.951442 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951281    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 13:11:23.951442 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 13:11:23.951672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 13:11:23.951672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951556    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 13:11:23.951971 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.951951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 13:11:23.952113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.952097    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zmcc2\""
Sep 16 13:11:23.953807 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.953738    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 13:11:23.956792 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.956771    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 13:11:23.966311 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.966290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.995458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.995887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.995934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.995970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90fdff09-1803-4a13-a5fd-df75392c9a8d-config-out\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-web-config\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4468j\" (UniqueName: \"kubernetes.io/projected/90fdff09-1803-4a13-a5fd-df75392c9a8d-kube-api-access-4468j\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000970 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-config\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000970 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000970 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90fdff09-1803-4a13-a5fd-df75392c9a8d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.000970 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:23.996844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098184 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098184 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90fdff09-1803-4a13-a5fd-df75392c9a8d-config-out\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-web-config\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4468j\" (UniqueName: \"kubernetes.io/projected/90fdff09-1803-4a13-a5fd-df75392c9a8d-kube-api-access-4468j\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-config\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.098535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90fdff09-1803-4a13-a5fd-df75392c9a8d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.098968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.099011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099527 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.099494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.099590 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.099527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.100191 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.100163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.100985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.100958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.101672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.101219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90fdff09-1803-4a13-a5fd-df75392c9a8d-config-out\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.101672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.101505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.101672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.101582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.101930 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.101908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.101991 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.101948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-web-config\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.102456 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.102426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.102804 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.102772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90fdff09-1803-4a13-a5fd-df75392c9a8d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.102946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.102807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90fdff09-1803-4a13-a5fd-df75392c9a8d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.103567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.103523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.103691 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.103673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.104257 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.104235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.104646 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.104624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-config\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.104686 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.104673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90fdff09-1803-4a13-a5fd-df75392c9a8d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.106710 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.106692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4468j\" (UniqueName: \"kubernetes.io/projected/90fdff09-1803-4a13-a5fd-df75392c9a8d-kube-api-access-4468j\") pod \"prometheus-k8s-0\" (UID: \"90fdff09-1803-4a13-a5fd-df75392c9a8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.258742 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.258693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:11:24.545363 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.545340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:11:24.547123 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:11:24.547098    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fdff09_1803_4a13_a5fd_df75392c9a8d.slice/crio-07e0e8cc964ceb2e849a1be746d62e21da686b485a890833204223e369496ce7 WatchSource:0}: Error finding container 07e0e8cc964ceb2e849a1be746d62e21da686b485a890833204223e369496ce7: Status 404 returned error can't find the container with id 07e0e8cc964ceb2e849a1be746d62e21da686b485a890833204223e369496ce7
Sep 16 13:11:24.891828 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.891736    2577 generic.go:356] "Generic (PLEG): container finished" podID="90fdff09-1803-4a13-a5fd-df75392c9a8d" containerID="8933f8fe0da3986533bf104eb46dcfeabd3f8bd694acb6d3fc8ad65a2a957553" exitCode=0
Sep 16 13:11:24.891828 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.891793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90fdff09-1803-4a13-a5fd-df75392c9a8d","Type":"ContainerDied","Data":"8933f8fe0da3986533bf104eb46dcfeabd3f8bd694acb6d3fc8ad65a2a957553"}
Sep 16 13:11:24.892496 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:24.891893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90fdff09-1803-4a13-a5fd-df75392c9a8d","Type":"ContainerStarted","Data":"07e0e8cc964ceb2e849a1be746d62e21da686b485a890833204223e369496ce7"}
Sep 16 13:11:25.116331 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:25.116300    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b7445c4-3539-4e5d-81e7-aeef6e19e1ed" path="/var/lib/kubelet/pods/6b7445c4-3539-4e5d-81e7-aeef6e19e1ed/volumes"
Sep 16 13:11:25.899536 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:25.899503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90fdff09-1803-4a13-a5fd-df75392c9a8d","Type":"ContainerStarted","Data":"68c26ad6d1a5eb1c2660b310a5a35b76c21ff630b3f12a2168342516a2a8ea1b"}
Sep 16 13:11:25.900019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:25.899567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90fdff09-1803-4a13-a5fd-df75392c9a8d","Type":"ContainerStarted","Data":"02d0e9e5de8fac41c36637a32c88b89d714d998fe3e1faa475eae7012262106c"}
Sep 16 13:11:25.900019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:25.899598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90fdff09-1803-4a13-a5fd-df75392c9a8d","Type":"ContainerStarted","Data":"10a3955c0bc17d0fef472a9daea95b885f2f2078a1057910796b057ffea426d8"}
Sep 16 13:11:25.900019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:25.899624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90fdff09-1803-4a13-a5fd-df75392c9a8d","Type":"ContainerStarted","Data":"8249f0e958ac7fc6f58b0d3eaf175ec664bcc097caedccf3ef7106114199d834"}
Sep 16 13:11:25.900019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:25.899648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90fdff09-1803-4a13-a5fd-df75392c9a8d","Type":"ContainerStarted","Data":"f92b7301f4199e6f80cefcf9ef3b95c6017843db63cdeaae4298ec008965195f"}
Sep 16 13:11:25.900019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:25.899670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90fdff09-1803-4a13-a5fd-df75392c9a8d","Type":"ContainerStarted","Data":"b6070f1525f89980c207f3a1419eff82a02d44785fbf24e58cd295ff0a7b9834"}
Sep 16 13:11:25.926499 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:25.926436    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.908342866 podStartE2EDuration="2.926422666s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.0798ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:11:25.926032055 +0000 UTC m=+157.348773634" watchObservedRunningTime="2026-09-16 13:11:25.926422666 +0000 UTC m=+157.349164287"
Sep 16 13:11:29.259632 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:11:29.259581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:12:24.259359 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:12:24.259312    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:12:24.275312 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:12:24.275285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:12:25.154901 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:12:25.154874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:13:13.058956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.058874    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4zrn7"]
Sep 16 13:13:13.061380 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.061365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:13.064030 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.064010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sdkrm\""
Sep 16 13:13:13.064139 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.064095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 13:13:13.064715 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.064694    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 13:13:13.078590 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.078563    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4zrn7"]
Sep 16 13:13:13.126026 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.125992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f63b88f-abe9-4086-8396-15fed2cc34f9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4zrn7\" (UID: \"5f63b88f-abe9-4086-8396-15fed2cc34f9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:13.126197 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.126052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4sk\" (UniqueName: \"kubernetes.io/projected/5f63b88f-abe9-4086-8396-15fed2cc34f9-kube-api-access-dw4sk\") pod \"cert-manager-webhook-6486ffd997-4zrn7\" (UID: \"5f63b88f-abe9-4086-8396-15fed2cc34f9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:13.226682 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.226646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f63b88f-abe9-4086-8396-15fed2cc34f9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4zrn7\" (UID: \"5f63b88f-abe9-4086-8396-15fed2cc34f9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:13.226877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.226712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4sk\" (UniqueName: \"kubernetes.io/projected/5f63b88f-abe9-4086-8396-15fed2cc34f9-kube-api-access-dw4sk\") pod \"cert-manager-webhook-6486ffd997-4zrn7\" (UID: \"5f63b88f-abe9-4086-8396-15fed2cc34f9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:13.245544 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.245508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f63b88f-abe9-4086-8396-15fed2cc34f9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4zrn7\" (UID: \"5f63b88f-abe9-4086-8396-15fed2cc34f9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:13.245661 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.245573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4sk\" (UniqueName: \"kubernetes.io/projected/5f63b88f-abe9-4086-8396-15fed2cc34f9-kube-api-access-dw4sk\") pod \"cert-manager-webhook-6486ffd997-4zrn7\" (UID: \"5f63b88f-abe9-4086-8396-15fed2cc34f9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:13.386319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.386231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:13.674912 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:13.674822    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4zrn7"]
Sep 16 13:13:13.678305 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:13:13.678276    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f63b88f_abe9_4086_8396_15fed2cc34f9.slice/crio-854e61fea757fff21854e910acb18c4e1476ac097c067c03e3cc83364e768de5 WatchSource:0}: Error finding container 854e61fea757fff21854e910acb18c4e1476ac097c067c03e3cc83364e768de5: Status 404 returned error can't find the container with id 854e61fea757fff21854e910acb18c4e1476ac097c067c03e3cc83364e768de5
Sep 16 13:13:14.342411 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:14.342373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7" event={"ID":"5f63b88f-abe9-4086-8396-15fed2cc34f9","Type":"ContainerStarted","Data":"854e61fea757fff21854e910acb18c4e1476ac097c067c03e3cc83364e768de5"}
Sep 16 13:13:15.496951 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.496917    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"]
Sep 16 13:13:15.503918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.503888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"
Sep 16 13:13:15.506879 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.506830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mf6kk\""
Sep 16 13:13:15.513244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.513191    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"]
Sep 16 13:13:15.549288 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.549246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fjkf\" (UniqueName: \"kubernetes.io/projected/e5bd52b3-d607-4c6b-a055-9c3bdc1265d8-kube-api-access-6fjkf\") pod \"cert-manager-cainjector-559cdd8c87-8kqhf\" (UID: \"e5bd52b3-d607-4c6b-a055-9c3bdc1265d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"
Sep 16 13:13:15.549512 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.549492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5bd52b3-d607-4c6b-a055-9c3bdc1265d8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8kqhf\" (UID: \"e5bd52b3-d607-4c6b-a055-9c3bdc1265d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"
Sep 16 13:13:15.650227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.650189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5bd52b3-d607-4c6b-a055-9c3bdc1265d8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8kqhf\" (UID: \"e5bd52b3-d607-4c6b-a055-9c3bdc1265d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"
Sep 16 13:13:15.650435 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.650284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjkf\" (UniqueName: \"kubernetes.io/projected/e5bd52b3-d607-4c6b-a055-9c3bdc1265d8-kube-api-access-6fjkf\") pod \"cert-manager-cainjector-559cdd8c87-8kqhf\" (UID: \"e5bd52b3-d607-4c6b-a055-9c3bdc1265d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"
Sep 16 13:13:15.658771 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.658747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5bd52b3-d607-4c6b-a055-9c3bdc1265d8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8kqhf\" (UID: \"e5bd52b3-d607-4c6b-a055-9c3bdc1265d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"
Sep 16 13:13:15.658942 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.658785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjkf\" (UniqueName: \"kubernetes.io/projected/e5bd52b3-d607-4c6b-a055-9c3bdc1265d8-kube-api-access-6fjkf\") pod \"cert-manager-cainjector-559cdd8c87-8kqhf\" (UID: \"e5bd52b3-d607-4c6b-a055-9c3bdc1265d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"
Sep 16 13:13:15.818957 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:15.818846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"
Sep 16 13:13:17.093809 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:17.093787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf"]
Sep 16 13:13:17.096005 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:13:17.095976    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5bd52b3_d607_4c6b_a055_9c3bdc1265d8.slice/crio-180ea8a0381d6bc6cbd714c46235cc86e1d2662435744336eb774ed75e4e3af2 WatchSource:0}: Error finding container 180ea8a0381d6bc6cbd714c46235cc86e1d2662435744336eb774ed75e4e3af2: Status 404 returned error can't find the container with id 180ea8a0381d6bc6cbd714c46235cc86e1d2662435744336eb774ed75e4e3af2
Sep 16 13:13:17.360644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:17.360535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7" event={"ID":"5f63b88f-abe9-4086-8396-15fed2cc34f9","Type":"ContainerStarted","Data":"5a192d78017bc24eb5fcf96b261c5546290df3f05570391f7eb3066acce8ec1c"}
Sep 16 13:13:17.360644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:17.360595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:17.362255 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:17.362225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf" event={"ID":"e5bd52b3-d607-4c6b-a055-9c3bdc1265d8","Type":"ContainerStarted","Data":"aad6df6ac625d044d65ecfb086a81c1d56b366498e020a2eb91509e77c32d1b0"}
Sep 16 13:13:17.362386 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:17.362262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf" event={"ID":"e5bd52b3-d607-4c6b-a055-9c3bdc1265d8","Type":"ContainerStarted","Data":"180ea8a0381d6bc6cbd714c46235cc86e1d2662435744336eb774ed75e4e3af2"}
Sep 16 13:13:17.394873 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:17.388738    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7" podStartSLOduration=1.208477737 podStartE2EDuration="4.388715603s" totalImagesPullingTime="3.180237866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:17.386755505 +0000 UTC m=+268.809497084" watchObservedRunningTime="2026-09-16 13:13:17.388715603 +0000 UTC m=+268.811457183"
Sep 16 13:13:17.411559 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:17.411496    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8kqhf" podStartSLOduration=2.411472957 podStartE2EDuration="2.411472957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:17.410104034 +0000 UTC m=+268.832845611" watchObservedRunningTime="2026-09-16 13:13:17.411472957 +0000 UTC m=+268.834214538"
Sep 16 13:13:23.369061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:23.369027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4zrn7"
Sep 16 13:13:48.986674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:48.986650    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 13:13:51.378514 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.378468    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"]
Sep 16 13:13:51.381268 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.381252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.391592 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.391572    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 13:13:51.392095 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.392073    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 13:13:51.392758 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.392742    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qxzqt\""
Sep 16 13:13:51.399306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.399290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 13:13:51.406326 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.406301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 13:13:51.431274 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.431244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"]
Sep 16 13:13:51.479377 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.479347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e49a2d2-a9f6-450d-8432-27aff205dde4-tmp\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.479565 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.479449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e49a2d2-a9f6-450d-8432-27aff205dde4-manifests\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.479565 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.479485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e49a2d2-a9f6-450d-8432-27aff205dde4-webhook-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.479565 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.479519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4hp7\" (UniqueName: \"kubernetes.io/projected/6e49a2d2-a9f6-450d-8432-27aff205dde4-kube-api-access-r4hp7\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.479720 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.479568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e49a2d2-a9f6-450d-8432-27aff205dde4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.580930 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.580897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e49a2d2-a9f6-450d-8432-27aff205dde4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.580930 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.580941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e49a2d2-a9f6-450d-8432-27aff205dde4-tmp\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.581174 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.580997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e49a2d2-a9f6-450d-8432-27aff205dde4-manifests\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.581174 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.581129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e49a2d2-a9f6-450d-8432-27aff205dde4-webhook-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.581254 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.581168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hp7\" (UniqueName: \"kubernetes.io/projected/6e49a2d2-a9f6-450d-8432-27aff205dde4-kube-api-access-r4hp7\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.581383 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.581362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e49a2d2-a9f6-450d-8432-27aff205dde4-manifests\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.583327 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.583298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e49a2d2-a9f6-450d-8432-27aff205dde4-tmp\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.583467 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.583426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e49a2d2-a9f6-450d-8432-27aff205dde4-webhook-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.583568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.583551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e49a2d2-a9f6-450d-8432-27aff205dde4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.593345 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.593317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hp7\" (UniqueName: \"kubernetes.io/projected/6e49a2d2-a9f6-450d-8432-27aff205dde4-kube-api-access-r4hp7\") pod \"opendatahub-operator-controller-manager-694ff5d95c-w2j56\" (UID: \"6e49a2d2-a9f6-450d-8432-27aff205dde4\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.690933 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.690905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:51.980939 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.980839    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"]
Sep 16 13:13:51.984086 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:13:51.984042    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e49a2d2_a9f6_450d_8432_27aff205dde4.slice/crio-3c8ea6886fb44ed2b4bb6ef1117c6f654ad562e9258457977b187114c294e311 WatchSource:0}: Error finding container 3c8ea6886fb44ed2b4bb6ef1117c6f654ad562e9258457977b187114c294e311: Status 404 returned error can't find the container with id 3c8ea6886fb44ed2b4bb6ef1117c6f654ad562e9258457977b187114c294e311
Sep 16 13:13:51.985770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:51.985748    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:13:52.502629 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:52.502585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56" event={"ID":"6e49a2d2-a9f6-450d-8432-27aff205dde4","Type":"ContainerStarted","Data":"3c8ea6886fb44ed2b4bb6ef1117c6f654ad562e9258457977b187114c294e311"}
Sep 16 13:13:55.519505 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:55.519472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56" event={"ID":"6e49a2d2-a9f6-450d-8432-27aff205dde4","Type":"ContainerStarted","Data":"e9539786729540839676a9c81e7e1aaabafa34ee4d76ac3f336bcf5865f346ca"}
Sep 16 13:13:56.525555 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:56.525461    2577 generic.go:356] "Generic (PLEG): container finished" podID="6e49a2d2-a9f6-450d-8432-27aff205dde4" containerID="e9539786729540839676a9c81e7e1aaabafa34ee4d76ac3f336bcf5865f346ca" exitCode=0
Sep 16 13:13:56.525939 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:56.525549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56" event={"ID":"6e49a2d2-a9f6-450d-8432-27aff205dde4","Type":"ContainerDied","Data":"e9539786729540839676a9c81e7e1aaabafa34ee4d76ac3f336bcf5865f346ca"}
Sep 16 13:13:57.532017 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:57.531981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56" event={"ID":"6e49a2d2-a9f6-450d-8432-27aff205dde4","Type":"ContainerStarted","Data":"02cecd42ef2cb269100ee05429e638134a4613c94e7566f4367f5cba792b001d"}
Sep 16 13:13:57.532488 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:57.532082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:13:57.573687 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:13:57.573624    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56" podStartSLOduration=1.9553894710000002 podStartE2EDuration="6.573601444s" totalImagesPullingTime="3.53704243s" totalInitContainerRuntime="1.081169543s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:57.571401819 +0000 UTC m=+308.994143409" watchObservedRunningTime="2026-09-16 13:13:57.573601444 +0000 UTC m=+308.996343024"
Sep 16 13:14:08.539262 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:08.539228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-w2j56"
Sep 16 13:14:18.432895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.432846    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"]
Sep 16 13:14:18.435740 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.435714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.438430 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.438405    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:14:18.438581 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.438470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-2qvlv\""
Sep 16 13:14:18.438581 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.438511    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 13:14:18.439336 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.439319    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 13:14:18.451212 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.451181    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"]
Sep 16 13:14:18.529745 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.529708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.529745 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.529747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.529995 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.529775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.529995 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.529843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxzk\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-kube-api-access-wdxzk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.529995 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.529917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.529995 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.529948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18388b35-0739-48d1-b004-24b5417c8614-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.530125 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.529996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.530125 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.530012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.530125 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.530045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18388b35-0739-48d1-b004-24b5417c8614-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631104 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631104 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18388b35-0739-48d1-b004-24b5417c8614-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18388b35-0739-48d1-b004-24b5417c8614-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxzk\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-kube-api-access-wdxzk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631689 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631689 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631791 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.631903 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.631879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.632039 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.632018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18388b35-0739-48d1-b004-24b5417c8614-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.633948 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.633927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.634333 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.634311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18388b35-0739-48d1-b004-24b5417c8614-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.657705 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.657661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.657887 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.657827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxzk\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-kube-api-access-wdxzk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcngcqq\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:18.748798 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:18.748756    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:19.029795 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:19.029716    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"]
Sep 16 13:14:19.033000 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:14:19.032975    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18388b35_0739_48d1_b004_24b5417c8614.slice/crio-757fd29a174bdbc790597b4b51d87a6af2434391d18a011e273a662153be0178 WatchSource:0}: Error finding container 757fd29a174bdbc790597b4b51d87a6af2434391d18a011e273a662153be0178: Status 404 returned error can't find the container with id 757fd29a174bdbc790597b4b51d87a6af2434391d18a011e273a662153be0178
Sep 16 13:14:19.635775 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:19.635733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" event={"ID":"18388b35-0739-48d1-b004-24b5417c8614","Type":"ContainerStarted","Data":"757fd29a174bdbc790597b4b51d87a6af2434391d18a011e273a662153be0178"}
Sep 16 13:14:21.679526 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:21.679480    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:14:21.679928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:21.679566    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:14:21.679928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:21.679615    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:14:22.653844 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:22.653800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" event={"ID":"18388b35-0739-48d1-b004-24b5417c8614","Type":"ContainerStarted","Data":"a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a"}
Sep 16 13:14:22.676045 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:22.675989    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" podStartSLOduration=2.031663208 podStartE2EDuration="4.675971553s" totalImagesPullingTime="2.644308345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:22.67355717 +0000 UTC m=+334.096298764" watchObservedRunningTime="2026-09-16 13:14:22.675971553 +0000 UTC m=+334.098713133"
Sep 16 13:14:22.749067 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:22.749029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:22.750449 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:22.750414    2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 16 13:14:22.750567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:22.750475    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" podUID="18388b35-0739-48d1-b004-24b5417c8614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 16 13:14:23.749496 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:23.749419    2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 16 13:14:23.749978 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:23.749574    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" podUID="18388b35-0739-48d1-b004-24b5417c8614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 16 13:14:23.928134 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:23.928097    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"]
Sep 16 13:14:24.749498 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:24.749451    2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 16 13:14:24.749991 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:24.749517    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" podUID="18388b35-0739-48d1-b004-24b5417c8614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 16 13:14:25.667619 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:25.667581    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" podUID="18388b35-0739-48d1-b004-24b5417c8614" containerName="istio-proxy" containerID="cri-o://a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a" gracePeriod=30
Sep 16 13:14:29.718040 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.717989    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"]
Sep 16 13:14:29.721928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.721905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.725799 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.725778    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 13:14:29.726044 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.726024    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:14:29.726585 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.726562    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 13:14:29.726672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.726592    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 13:14:29.726672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.726562    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2g85n\""
Sep 16 13:14:29.726996 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.726976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 13:14:29.740358 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.740326    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"]
Sep 16 13:14:29.832347 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.832303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1da303e1-7f13-4600-bf42-18ab63b5d09b-metrics-cert\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.832586 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.832388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1da303e1-7f13-4600-bf42-18ab63b5d09b-manager-config\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.832586 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.832408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1da303e1-7f13-4600-bf42-18ab63b5d09b-cert\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.832586 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.832437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxt7\" (UniqueName: \"kubernetes.io/projected/1da303e1-7f13-4600-bf42-18ab63b5d09b-kube-api-access-qkxt7\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.932865 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.932818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1da303e1-7f13-4600-bf42-18ab63b5d09b-manager-config\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.933064 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.932876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1da303e1-7f13-4600-bf42-18ab63b5d09b-cert\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.933064 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.932991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxt7\" (UniqueName: \"kubernetes.io/projected/1da303e1-7f13-4600-bf42-18ab63b5d09b-kube-api-access-qkxt7\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.933179 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.933120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1da303e1-7f13-4600-bf42-18ab63b5d09b-metrics-cert\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.933567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.933541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1da303e1-7f13-4600-bf42-18ab63b5d09b-manager-config\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.935359 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.935340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1da303e1-7f13-4600-bf42-18ab63b5d09b-metrics-cert\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.935411 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.935339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1da303e1-7f13-4600-bf42-18ab63b5d09b-cert\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:29.941729 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:29.941702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxt7\" (UniqueName: \"kubernetes.io/projected/1da303e1-7f13-4600-bf42-18ab63b5d09b-kube-api-access-qkxt7\") pod \"lws-controller-manager-5b97dbbc45-85rrr\" (UID: \"1da303e1-7f13-4600-bf42-18ab63b5d09b\") " pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:30.036013 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.035917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:30.330804 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.330771    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"]
Sep 16 13:14:30.333472 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:14:30.333447    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da303e1_7f13_4600_bf42_18ab63b5d09b.slice/crio-3e8b06ddac8b6085d252f27e1e03cd025dccad4410cf90c0bf70463e040091e1 WatchSource:0}: Error finding container 3e8b06ddac8b6085d252f27e1e03cd025dccad4410cf90c0bf70463e040091e1: Status 404 returned error can't find the container with id 3e8b06ddac8b6085d252f27e1e03cd025dccad4410cf90c0bf70463e040091e1
Sep 16 13:14:30.692520 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.692489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr" event={"ID":"1da303e1-7f13-4600-bf42-18ab63b5d09b","Type":"ContainerStarted","Data":"3e8b06ddac8b6085d252f27e1e03cd025dccad4410cf90c0bf70463e040091e1"}
Sep 16 13:14:30.896703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.896678    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:30.945986 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.945960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18388b35-0739-48d1-b004-24b5417c8614-istio-podinfo\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946126 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.945996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-socket\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946126 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946032    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-credential-socket\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946239 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-envoy\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946239 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-data\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946338 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-certs\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946338 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18388b35-0739-48d1-b004-24b5417c8614-istiod-ca-cert\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946338 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946310    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdxzk\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-kube-api-access-wdxzk\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946338 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:14:30.946547 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-istio-token\") pod \"18388b35-0739-48d1-b004-24b5417c8614\" (UID: \"18388b35-0739-48d1-b004-24b5417c8614\") "
Sep 16 13:14:30.946547 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:14:30.946547 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-data" (OuterVolumeSpecName: "istio-data") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:14:30.946547 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946519    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:14:30.946747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946625    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18388b35-0739-48d1-b004-24b5417c8614-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:14:30.946747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946710    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:30.946747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946730    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:30.946747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946745    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18388b35-0739-48d1-b004-24b5417c8614-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:30.946962 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946758    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:30.946962 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.946771    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:30.948417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.948386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:14:30.948524 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.948453    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-kube-api-access-wdxzk" (OuterVolumeSpecName: "kube-api-access-wdxzk") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "kube-api-access-wdxzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:14:30.948524 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.948475    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18388b35-0739-48d1-b004-24b5417c8614-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:14:30.948524 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:30.948500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-istio-token" (OuterVolumeSpecName: "istio-token") pod "18388b35-0739-48d1-b004-24b5417c8614" (UID: "18388b35-0739-48d1-b004-24b5417c8614"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:14:31.048160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.048127    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18388b35-0739-48d1-b004-24b5417c8614-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:31.048160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.048154    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdxzk\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-kube-api-access-wdxzk\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:31.048160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.048165    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18388b35-0739-48d1-b004-24b5417c8614-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:31.048392 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.048174    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18388b35-0739-48d1-b004-24b5417c8614-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:14:31.465628 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.465593    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"]
Sep 16 13:14:31.466424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.466139    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18388b35-0739-48d1-b004-24b5417c8614" containerName="istio-proxy"
Sep 16 13:14:31.466424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.466161    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18388b35-0739-48d1-b004-24b5417c8614" containerName="istio-proxy"
Sep 16 13:14:31.466424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.466251    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18388b35-0739-48d1-b004-24b5417c8614" containerName="istio-proxy"
Sep 16 13:14:31.468731 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.468706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:31.471339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.471317    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 13:14:31.471435 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.471397    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6q8fj\""
Sep 16 13:14:31.476813 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.476792    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"]
Sep 16 13:14:31.482372 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.482341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:31.482546 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.482351    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"]
Sep 16 13:14:31.485055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.484872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 13:14:31.485166 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.485074    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-w7tzn\""
Sep 16 13:14:31.490782 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.490762    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"]
Sep 16 13:14:31.552408 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.552376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwtp\" (UniqueName: \"kubernetes.io/projected/0e58e521-b04f-4495-8451-b7e949b23400-kube-api-access-7pwtp\") pod \"kserve-controller-manager-d6f4b65b8-nfx77\" (UID: \"0e58e521-b04f-4495-8451-b7e949b23400\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:31.552593 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.552490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14c58d86-e25f-4d4a-bcac-cd5d5d672adb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hggtb\" (UID: \"14c58d86-e25f-4d4a-bcac-cd5d5d672adb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:31.552593 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.552566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e58e521-b04f-4495-8451-b7e949b23400-cert\") pod \"kserve-controller-manager-d6f4b65b8-nfx77\" (UID: \"0e58e521-b04f-4495-8451-b7e949b23400\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:31.552723 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.552658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zx4\" (UniqueName: \"kubernetes.io/projected/14c58d86-e25f-4d4a-bcac-cd5d5d672adb-kube-api-access-54zx4\") pod \"llmisvc-controller-manager-777c6bd7d5-hggtb\" (UID: \"14c58d86-e25f-4d4a-bcac-cd5d5d672adb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:31.654050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.654014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14c58d86-e25f-4d4a-bcac-cd5d5d672adb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hggtb\" (UID: \"14c58d86-e25f-4d4a-bcac-cd5d5d672adb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:31.654240 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.654090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e58e521-b04f-4495-8451-b7e949b23400-cert\") pod \"kserve-controller-manager-d6f4b65b8-nfx77\" (UID: \"0e58e521-b04f-4495-8451-b7e949b23400\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:31.654240 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.654125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zx4\" (UniqueName: \"kubernetes.io/projected/14c58d86-e25f-4d4a-bcac-cd5d5d672adb-kube-api-access-54zx4\") pod \"llmisvc-controller-manager-777c6bd7d5-hggtb\" (UID: \"14c58d86-e25f-4d4a-bcac-cd5d5d672adb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:31.654240 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.654173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwtp\" (UniqueName: \"kubernetes.io/projected/0e58e521-b04f-4495-8451-b7e949b23400-kube-api-access-7pwtp\") pod \"kserve-controller-manager-d6f4b65b8-nfx77\" (UID: \"0e58e521-b04f-4495-8451-b7e949b23400\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:31.654240 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:14:31.654207    2577 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 13:14:31.654454 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:14:31.654287    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e58e521-b04f-4495-8451-b7e949b23400-cert podName:0e58e521-b04f-4495-8451-b7e949b23400 nodeName:}" failed. No retries permitted until 2026-09-16 13:14:32.154262669 +0000 UTC m=+343.577004245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0e58e521-b04f-4495-8451-b7e949b23400-cert") pod "kserve-controller-manager-d6f4b65b8-nfx77" (UID: "0e58e521-b04f-4495-8451-b7e949b23400") : secret "kserve-webhook-server-cert" not found
Sep 16 13:14:31.657293 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.657261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14c58d86-e25f-4d4a-bcac-cd5d5d672adb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hggtb\" (UID: \"14c58d86-e25f-4d4a-bcac-cd5d5d672adb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:31.668804 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.668775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwtp\" (UniqueName: \"kubernetes.io/projected/0e58e521-b04f-4495-8451-b7e949b23400-kube-api-access-7pwtp\") pod \"kserve-controller-manager-d6f4b65b8-nfx77\" (UID: \"0e58e521-b04f-4495-8451-b7e949b23400\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:31.669174 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.669150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zx4\" (UniqueName: \"kubernetes.io/projected/14c58d86-e25f-4d4a-bcac-cd5d5d672adb-kube-api-access-54zx4\") pod \"llmisvc-controller-manager-777c6bd7d5-hggtb\" (UID: \"14c58d86-e25f-4d4a-bcac-cd5d5d672adb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:31.701686 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.701648    2577 generic.go:356] "Generic (PLEG): container finished" podID="18388b35-0739-48d1-b004-24b5417c8614" containerID="a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a" exitCode=0
Sep 16 13:14:31.701845 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.701757    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" event={"ID":"18388b35-0739-48d1-b004-24b5417c8614","Type":"ContainerDied","Data":"a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a"}
Sep 16 13:14:31.701845 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.701807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq" event={"ID":"18388b35-0739-48d1-b004-24b5417c8614","Type":"ContainerDied","Data":"757fd29a174bdbc790597b4b51d87a6af2434391d18a011e273a662153be0178"}
Sep 16 13:14:31.701845 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.701835    2577 scope.go:117] "RemoveContainer" containerID="a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a"
Sep 16 13:14:31.702037 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.702016    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"
Sep 16 13:14:31.724796 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.724772    2577 scope.go:117] "RemoveContainer" containerID="a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a"
Sep 16 13:14:31.726524 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:14:31.726489    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a\": container with ID starting with a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a not found: ID does not exist" containerID="a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a"
Sep 16 13:14:31.726637 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.726528    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a"} err="failed to get container status \"a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a\": rpc error: code = NotFound desc = could not find container \"a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a\": container with ID starting with a6be4143f9bc50ddb3d6f4b1bda1e64409414e19f667e4af31e992b3b8d2a22a not found: ID does not exist"
Sep 16 13:14:31.736026 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.735965    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"]
Sep 16 13:14:31.762813 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.762764    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcngcqq"]
Sep 16 13:14:31.800446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:31.799055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:32.083535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:32.083369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"]
Sep 16 13:14:32.159285 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:32.159248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e58e521-b04f-4495-8451-b7e949b23400-cert\") pod \"kserve-controller-manager-d6f4b65b8-nfx77\" (UID: \"0e58e521-b04f-4495-8451-b7e949b23400\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:32.161919 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:32.161892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e58e521-b04f-4495-8451-b7e949b23400-cert\") pod \"kserve-controller-manager-d6f4b65b8-nfx77\" (UID: \"0e58e521-b04f-4495-8451-b7e949b23400\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:32.389946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:32.389865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:32.747612 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:14:32.747582    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14c58d86_e25f_4d4a_bcac_cd5d5d672adb.slice/crio-aed40de1a61612ebe8a1c8ebfa4337e0683f3496430b5a8943ca2d1d96031a9d WatchSource:0}: Error finding container aed40de1a61612ebe8a1c8ebfa4337e0683f3496430b5a8943ca2d1d96031a9d: Status 404 returned error can't find the container with id aed40de1a61612ebe8a1c8ebfa4337e0683f3496430b5a8943ca2d1d96031a9d
Sep 16 13:14:33.109152 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:33.109080    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18388b35-0739-48d1-b004-24b5417c8614" path="/var/lib/kubelet/pods/18388b35-0739-48d1-b004-24b5417c8614/volumes"
Sep 16 13:14:33.396825 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:33.396788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"]
Sep 16 13:14:33.716045 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:33.716007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb" event={"ID":"14c58d86-e25f-4d4a-bcac-cd5d5d672adb","Type":"ContainerStarted","Data":"aed40de1a61612ebe8a1c8ebfa4337e0683f3496430b5a8943ca2d1d96031a9d"}
Sep 16 13:14:33.717785 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:33.717698    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr" event={"ID":"1da303e1-7f13-4600-bf42-18ab63b5d09b","Type":"ContainerStarted","Data":"447eac9e23cf06fabff215705a8615dbd7a51538d813e4f53a90fc4e4db8d345"}
Sep 16 13:14:33.717785 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:33.717756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:14:33.718894 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:33.718873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77" event={"ID":"0e58e521-b04f-4495-8451-b7e949b23400","Type":"ContainerStarted","Data":"8ead0b4fbba800331c83eac232b90fd00273c2f125e07c0ccf75819327332a55"}
Sep 16 13:14:33.750282 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:33.750232    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr" podStartSLOduration=1.929833716 podStartE2EDuration="4.750214982s" totalImagesPullingTime="2.820381266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:33.747016786 +0000 UTC m=+345.169758365" watchObservedRunningTime="2026-09-16 13:14:33.750214982 +0000 UTC m=+345.172956560"
Sep 16 13:14:37.745030 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:37.744991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb" event={"ID":"14c58d86-e25f-4d4a-bcac-cd5d5d672adb","Type":"ContainerStarted","Data":"d3f968a1ebc26f22078ae7178f9ca12866758910bc592fcf9a42a63db95a49d2"}
Sep 16 13:14:37.745457 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:37.745058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:14:37.746564 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:37.746531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77" event={"ID":"0e58e521-b04f-4495-8451-b7e949b23400","Type":"ContainerStarted","Data":"030b9c824f8a3fcc69c2df8dcb798e6e2c2d16ab753f94e5d3932aa0553bf693"}
Sep 16 13:14:37.746682 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:37.746662    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:14:37.761362 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:37.761319    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb" podStartSLOduration=2.508645834 podStartE2EDuration="6.761303153s" totalImagesPullingTime="4.252657319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:37.759526214 +0000 UTC m=+349.182267794" watchObservedRunningTime="2026-09-16 13:14:37.761303153 +0000 UTC m=+349.184044733"
Sep 16 13:14:37.775376 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:37.775327    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77" podStartSLOduration=3.129142076 podStartE2EDuration="6.775308306s" totalImagesPullingTime="3.64616623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:37.774916332 +0000 UTC m=+349.197657911" watchObservedRunningTime="2026-09-16 13:14:37.775308306 +0000 UTC m=+349.198049885"
Sep 16 13:14:44.729459 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:14:44.729383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b97dbbc45-85rrr"
Sep 16 13:15:08.753636 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:08.753602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hggtb"
Sep 16 13:15:08.756787 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:08.756760    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nfx77"
Sep 16 13:15:41.228599 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.228564    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"]
Sep 16 13:15:41.231400 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.231382    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"
Sep 16 13:15:41.234690 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.234670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-p5jg8\""
Sep 16 13:15:41.235448 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.235415    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 13:15:41.235598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.235580    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 13:15:41.238523 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.238500    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 13:15:41.249315 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.249294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"]
Sep 16 13:15:41.396291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.396248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6fk\" (UniqueName: \"kubernetes.io/projected/a575b729-7dd7-4354-880a-d8d5f1d1b933-kube-api-access-zt6fk\") pod \"dns-operator-controller-manager-7b84cd7c9c-z4lhk\" (UID: \"a575b729-7dd7-4354-880a-d8d5f1d1b933\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"
Sep 16 13:15:41.497147 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.497060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6fk\" (UniqueName: \"kubernetes.io/projected/a575b729-7dd7-4354-880a-d8d5f1d1b933-kube-api-access-zt6fk\") pod \"dns-operator-controller-manager-7b84cd7c9c-z4lhk\" (UID: \"a575b729-7dd7-4354-880a-d8d5f1d1b933\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"
Sep 16 13:15:41.507120 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.507092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6fk\" (UniqueName: \"kubernetes.io/projected/a575b729-7dd7-4354-880a-d8d5f1d1b933-kube-api-access-zt6fk\") pod \"dns-operator-controller-manager-7b84cd7c9c-z4lhk\" (UID: \"a575b729-7dd7-4354-880a-d8d5f1d1b933\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"
Sep 16 13:15:41.541622 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.541584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"
Sep 16 13:15:41.832626 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:41.832603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"]
Sep 16 13:15:41.835148 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:15:41.835119    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda575b729_7dd7_4354_880a_d8d5f1d1b933.slice/crio-a424c037dec56c75ff4082679dd1abe1745d7d3097970f56ec73fcd30dc6f5e5 WatchSource:0}: Error finding container a424c037dec56c75ff4082679dd1abe1745d7d3097970f56ec73fcd30dc6f5e5: Status 404 returned error can't find the container with id a424c037dec56c75ff4082679dd1abe1745d7d3097970f56ec73fcd30dc6f5e5
Sep 16 13:15:42.042136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:42.042097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk" event={"ID":"a575b729-7dd7-4354-880a-d8d5f1d1b933","Type":"ContainerStarted","Data":"a424c037dec56c75ff4082679dd1abe1745d7d3097970f56ec73fcd30dc6f5e5"}
Sep 16 13:15:43.742383 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:43.742337    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-h9qts"]
Sep 16 13:15:43.749111 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:43.749089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-h9qts"
Sep 16 13:15:43.751688 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:43.751513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4n2g8\""
Sep 16 13:15:43.759908 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:43.759880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-h9qts"]
Sep 16 13:15:43.923653 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:43.923611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8ht\" (UniqueName: \"kubernetes.io/projected/c79c518f-b7b1-4152-bb0c-1bf5338ef94e-kube-api-access-4x8ht\") pod \"authorino-operator-76b478bd78-h9qts\" (UID: \"c79c518f-b7b1-4152-bb0c-1bf5338ef94e\") " pod="kuadrant-system/authorino-operator-76b478bd78-h9qts"
Sep 16 13:15:44.024312 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:44.024221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8ht\" (UniqueName: \"kubernetes.io/projected/c79c518f-b7b1-4152-bb0c-1bf5338ef94e-kube-api-access-4x8ht\") pod \"authorino-operator-76b478bd78-h9qts\" (UID: \"c79c518f-b7b1-4152-bb0c-1bf5338ef94e\") " pod="kuadrant-system/authorino-operator-76b478bd78-h9qts"
Sep 16 13:15:44.033157 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:44.033126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8ht\" (UniqueName: \"kubernetes.io/projected/c79c518f-b7b1-4152-bb0c-1bf5338ef94e-kube-api-access-4x8ht\") pod \"authorino-operator-76b478bd78-h9qts\" (UID: \"c79c518f-b7b1-4152-bb0c-1bf5338ef94e\") " pod="kuadrant-system/authorino-operator-76b478bd78-h9qts"
Sep 16 13:15:44.063364 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:44.063328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-h9qts"
Sep 16 13:15:44.575204 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:44.575175    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-h9qts"]
Sep 16 13:15:44.577741 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:15:44.577715    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79c518f_b7b1_4152_bb0c_1bf5338ef94e.slice/crio-c87461fad5135d15e455e8f0f90f35d7d90635b63cdb6b02e2423950d5459e35 WatchSource:0}: Error finding container c87461fad5135d15e455e8f0f90f35d7d90635b63cdb6b02e2423950d5459e35: Status 404 returned error can't find the container with id c87461fad5135d15e455e8f0f90f35d7d90635b63cdb6b02e2423950d5459e35
Sep 16 13:15:45.058974 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:45.058935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-h9qts" event={"ID":"c79c518f-b7b1-4152-bb0c-1bf5338ef94e","Type":"ContainerStarted","Data":"c87461fad5135d15e455e8f0f90f35d7d90635b63cdb6b02e2423950d5459e35"}
Sep 16 13:15:45.060592 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:45.060558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk" event={"ID":"a575b729-7dd7-4354-880a-d8d5f1d1b933","Type":"ContainerStarted","Data":"571440b8c384c1e940db4dc4b005dbe7cf2c12de22f4f4223b0d987803941264"}
Sep 16 13:15:45.060734 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:45.060697    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"
Sep 16 13:15:45.080999 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:45.080953    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk" podStartSLOduration=1.559366517 podStartE2EDuration="4.080937974s" totalImagesPullingTime="2.521571457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:45.078961635 +0000 UTC m=+416.501703215" watchObservedRunningTime="2026-09-16 13:15:45.080937974 +0000 UTC m=+416.503679553"
Sep 16 13:15:46.066992 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:46.066954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-h9qts" event={"ID":"c79c518f-b7b1-4152-bb0c-1bf5338ef94e","Type":"ContainerStarted","Data":"105c4754aca668a98d807088a60e6932cdaaf7c45baa4e0f42824d1955bf4288"}
Sep 16 13:15:46.085283 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:46.085231    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-h9qts" podStartSLOduration=1.750860318 podStartE2EDuration="3.085215489s" totalImagesPullingTime="1.334355171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:46.084012291 +0000 UTC m=+417.506753870" watchObservedRunningTime="2026-09-16 13:15:46.085215489 +0000 UTC m=+417.507957068"
Sep 16 13:15:47.071936 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:47.071906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-h9qts"
Sep 16 13:15:50.853533 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.853490    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"]
Sep 16 13:15:50.856473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.856450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:50.859397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.859376    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jmzkj\""
Sep 16 13:15:50.869362 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.869340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"]
Sep 16 13:15:50.889477 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.889444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7befb8-0bca-4b55-9771-7e56a3aaa507-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" (UID: \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:50.889685 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.889666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zhp\" (UniqueName: \"kubernetes.io/projected/5d7befb8-0bca-4b55-9771-7e56a3aaa507-kube-api-access-j6zhp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" (UID: \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:50.990394 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.990334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7befb8-0bca-4b55-9771-7e56a3aaa507-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" (UID: \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:50.990573 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.990421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zhp\" (UniqueName: \"kubernetes.io/projected/5d7befb8-0bca-4b55-9771-7e56a3aaa507-kube-api-access-j6zhp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" (UID: \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:50.990868 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.990827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7befb8-0bca-4b55-9771-7e56a3aaa507-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" (UID: \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:50.998883 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:50.998840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zhp\" (UniqueName: \"kubernetes.io/projected/5d7befb8-0bca-4b55-9771-7e56a3aaa507-kube-api-access-j6zhp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" (UID: \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:51.166802 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:51.166718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:51.450349 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:51.450327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"]
Sep 16 13:15:51.452645 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:15:51.452605    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7befb8_0bca_4b55_9771_7e56a3aaa507.slice/crio-3b1e94077d09c84640818613b0947eebd9b611e84e78cdd4589f9ec2d10282ba WatchSource:0}: Error finding container 3b1e94077d09c84640818613b0947eebd9b611e84e78cdd4589f9ec2d10282ba: Status 404 returned error can't find the container with id 3b1e94077d09c84640818613b0947eebd9b611e84e78cdd4589f9ec2d10282ba
Sep 16 13:15:52.096219 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:52.096182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j" event={"ID":"5d7befb8-0bca-4b55-9771-7e56a3aaa507","Type":"ContainerStarted","Data":"3b1e94077d09c84640818613b0947eebd9b611e84e78cdd4589f9ec2d10282ba"}
Sep 16 13:15:56.069535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:56.069505    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-z4lhk"
Sep 16 13:15:56.120741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:56.120699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j" event={"ID":"5d7befb8-0bca-4b55-9771-7e56a3aaa507","Type":"ContainerStarted","Data":"f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9"}
Sep 16 13:15:56.120941 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:56.120783    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:15:56.147314 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:56.147258    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j" podStartSLOduration=2.332188054 podStartE2EDuration="6.147237659s" totalImagesPullingTime="3.815049605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:56.146537931 +0000 UTC m=+427.569279522" watchObservedRunningTime="2026-09-16 13:15:56.147237659 +0000 UTC m=+427.569979240"
Sep 16 13:15:58.079000 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:15:58.078961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-h9qts"
Sep 16 13:16:07.128086 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:07.128058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:16:09.618864 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.618830    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"]
Sep 16 13:16:09.619288 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.619027    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" containerName="manager" containerID="cri-o://f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9" gracePeriod=2
Sep 16 13:16:09.628263 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.628238    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"]
Sep 16 13:16:09.633452 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:16:09.633412    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"5d7befb8-0bca-4b55-9771-7e56a3aaa507\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\": pods \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:16:09.643207 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.643178    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"]
Sep 16 13:16:09.643758 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.643741    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" containerName="manager"
Sep 16 13:16:09.643811 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.643763    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" containerName="manager"
Sep 16 13:16:09.643895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.643883    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" containerName="manager"
Sep 16 13:16:09.647323 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.647303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:09.649224 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:16:09.649192    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" is forbidden: User \"system:node:ip-10-0-134-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-164.ec2.internal' and this object" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:16:09.661048 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.661022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"]
Sep 16 13:16:09.765460 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.765424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdl7z\" (UniqueName: \"kubernetes.io/projected/78150451-e55d-4136-8c32-eff3f8158a8a-kube-api-access-vdl7z\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n75wl\" (UID: \"78150451-e55d-4136-8c32-eff3f8158a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:09.765621 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.765477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78150451-e55d-4136-8c32-eff3f8158a8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n75wl\" (UID: \"78150451-e55d-4136-8c32-eff3f8158a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:09.856982 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.856961    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:16:09.858841 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:16:09.858816    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" is forbidden: User \"system:node:ip-10-0-134-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-164.ec2.internal' and this object" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:16:09.866252 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.866229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdl7z\" (UniqueName: \"kubernetes.io/projected/78150451-e55d-4136-8c32-eff3f8158a8a-kube-api-access-vdl7z\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n75wl\" (UID: \"78150451-e55d-4136-8c32-eff3f8158a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:09.866334 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.866269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78150451-e55d-4136-8c32-eff3f8158a8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n75wl\" (UID: \"78150451-e55d-4136-8c32-eff3f8158a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:09.866619 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.866601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78150451-e55d-4136-8c32-eff3f8158a8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n75wl\" (UID: \"78150451-e55d-4136-8c32-eff3f8158a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:09.879513 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.879459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdl7z\" (UniqueName: \"kubernetes.io/projected/78150451-e55d-4136-8c32-eff3f8158a8a-kube-api-access-vdl7z\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n75wl\" (UID: \"78150451-e55d-4136-8c32-eff3f8158a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:09.966815 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.966781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6zhp\" (UniqueName: \"kubernetes.io/projected/5d7befb8-0bca-4b55-9771-7e56a3aaa507-kube-api-access-j6zhp\") pod \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\" (UID: \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\") "
Sep 16 13:16:09.966984 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.966845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7befb8-0bca-4b55-9771-7e56a3aaa507-extensions-socket-volume\") pod \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\" (UID: \"5d7befb8-0bca-4b55-9771-7e56a3aaa507\") "
Sep 16 13:16:09.967388 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.967363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7befb8-0bca-4b55-9771-7e56a3aaa507-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5d7befb8-0bca-4b55-9771-7e56a3aaa507" (UID: "5d7befb8-0bca-4b55-9771-7e56a3aaa507"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:09.968966 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:09.968938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7befb8-0bca-4b55-9771-7e56a3aaa507-kube-api-access-j6zhp" (OuterVolumeSpecName: "kube-api-access-j6zhp") pod "5d7befb8-0bca-4b55-9771-7e56a3aaa507" (UID: "5d7befb8-0bca-4b55-9771-7e56a3aaa507"). InnerVolumeSpecName "kube-api-access-j6zhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:10.000086 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.000056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:10.068488 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.068459    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6zhp\" (UniqueName: \"kubernetes.io/projected/5d7befb8-0bca-4b55-9771-7e56a3aaa507-kube-api-access-j6zhp\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:16:10.068488 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.068489    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7befb8-0bca-4b55-9771-7e56a3aaa507-extensions-socket-volume\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:16:10.190748 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.190718    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" containerID="f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9" exitCode=0
Sep 16 13:16:10.190914 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.190760    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:16:10.190914 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.190809    2577 scope.go:117] "RemoveContainer" containerID="f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9"
Sep 16 13:16:10.193080 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:16:10.193051    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" is forbidden: User \"system:node:ip-10-0-134-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-164.ec2.internal' and this object" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:16:10.199914 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.199844    2577 scope.go:117] "RemoveContainer" containerID="f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9"
Sep 16 13:16:10.200214 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:16:10.200198    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9\": container with ID starting with f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9 not found: ID does not exist" containerID="f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9"
Sep 16 13:16:10.200269 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.200224    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9"} err="failed to get container status \"f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9\": rpc error: code = NotFound desc = could not find container \"f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9\": container with ID starting with f97575f157f57fe7c0790bb4128e552f492d3eb886210e1c48a50bf8726648b9 not found: ID does not exist"
Sep 16 13:16:10.200982 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:16:10.200960    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zlw9j\" is forbidden: User \"system:node:ip-10-0-134-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-164.ec2.internal' and this object" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zlw9j"
Sep 16 13:16:10.285162 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:10.285121    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"]
Sep 16 13:16:10.289331 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:16:10.289301    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78150451_e55d_4136_8c32_eff3f8158a8a.slice/crio-d53c6da0518cb5ea70ef76c61b26d1c2dca972da60b357362ff2f08cee6064e8 WatchSource:0}: Error finding container d53c6da0518cb5ea70ef76c61b26d1c2dca972da60b357362ff2f08cee6064e8: Status 404 returned error can't find the container with id d53c6da0518cb5ea70ef76c61b26d1c2dca972da60b357362ff2f08cee6064e8
Sep 16 13:16:11.109938 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:11.109836    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7befb8-0bca-4b55-9771-7e56a3aaa507" path="/var/lib/kubelet/pods/5d7befb8-0bca-4b55-9771-7e56a3aaa507/volumes"
Sep 16 13:16:11.197891 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:11.197827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl" event={"ID":"78150451-e55d-4136-8c32-eff3f8158a8a","Type":"ContainerStarted","Data":"553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f"}
Sep 16 13:16:11.198055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:11.197912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl" event={"ID":"78150451-e55d-4136-8c32-eff3f8158a8a","Type":"ContainerStarted","Data":"d53c6da0518cb5ea70ef76c61b26d1c2dca972da60b357362ff2f08cee6064e8"}
Sep 16 13:16:11.198055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:11.197953    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:11.225969 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:11.225923    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl" podStartSLOduration=2.225907908 podStartE2EDuration="2.225907908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:11.223132998 +0000 UTC m=+442.645874575" watchObservedRunningTime="2026-09-16 13:16:11.225907908 +0000 UTC m=+442.648649546"
Sep 16 13:16:22.205458 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:22.205424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:26.528470 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:26.528436    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"]
Sep 16 13:16:26.528894 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:26.528650    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl" podUID="78150451-e55d-4136-8c32-eff3f8158a8a" containerName="manager" containerID="cri-o://553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f" gracePeriod=10
Sep 16 13:16:26.871653 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:26.871631    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:27.022928 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.022881    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdl7z\" (UniqueName: \"kubernetes.io/projected/78150451-e55d-4136-8c32-eff3f8158a8a-kube-api-access-vdl7z\") pod \"78150451-e55d-4136-8c32-eff3f8158a8a\" (UID: \"78150451-e55d-4136-8c32-eff3f8158a8a\") "
Sep 16 13:16:27.023103 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.022982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78150451-e55d-4136-8c32-eff3f8158a8a-extensions-socket-volume\") pod \"78150451-e55d-4136-8c32-eff3f8158a8a\" (UID: \"78150451-e55d-4136-8c32-eff3f8158a8a\") "
Sep 16 13:16:27.023349 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.023324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78150451-e55d-4136-8c32-eff3f8158a8a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "78150451-e55d-4136-8c32-eff3f8158a8a" (UID: "78150451-e55d-4136-8c32-eff3f8158a8a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:27.025014 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.024994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78150451-e55d-4136-8c32-eff3f8158a8a-kube-api-access-vdl7z" (OuterVolumeSpecName: "kube-api-access-vdl7z") pod "78150451-e55d-4136-8c32-eff3f8158a8a" (UID: "78150451-e55d-4136-8c32-eff3f8158a8a"). InnerVolumeSpecName "kube-api-access-vdl7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:27.123554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.123521    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78150451-e55d-4136-8c32-eff3f8158a8a-extensions-socket-volume\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:16:27.123554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.123547    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdl7z\" (UniqueName: \"kubernetes.io/projected/78150451-e55d-4136-8c32-eff3f8158a8a-kube-api-access-vdl7z\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:16:27.280102 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.280067    2577 generic.go:356] "Generic (PLEG): container finished" podID="78150451-e55d-4136-8c32-eff3f8158a8a" containerID="553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f" exitCode=0
Sep 16 13:16:27.280277 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.280110    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"
Sep 16 13:16:27.280277 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.280136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl" event={"ID":"78150451-e55d-4136-8c32-eff3f8158a8a","Type":"ContainerDied","Data":"553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f"}
Sep 16 13:16:27.280277 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.280183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl" event={"ID":"78150451-e55d-4136-8c32-eff3f8158a8a","Type":"ContainerDied","Data":"d53c6da0518cb5ea70ef76c61b26d1c2dca972da60b357362ff2f08cee6064e8"}
Sep 16 13:16:27.280277 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.280207    2577 scope.go:117] "RemoveContainer" containerID="553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f"
Sep 16 13:16:27.288228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.288207    2577 scope.go:117] "RemoveContainer" containerID="553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f"
Sep 16 13:16:27.288508 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:16:27.288482    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f\": container with ID starting with 553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f not found: ID does not exist" containerID="553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f"
Sep 16 13:16:27.288573 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.288511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f"} err="failed to get container status \"553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f\": rpc error: code = NotFound desc = could not find container \"553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f\": container with ID starting with 553ce3c26bb18ab8c4db29d198331cbc78f435764fda6e4fbf65a2842b9f324f not found: ID does not exist"
Sep 16 13:16:27.311488 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.311460    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"]
Sep 16 13:16:27.318265 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:27.318244    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n75wl"]
Sep 16 13:16:29.109722 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:29.109688    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78150451-e55d-4136-8c32-eff3f8158a8a" path="/var/lib/kubelet/pods/78150451-e55d-4136-8c32-eff3f8158a8a/volumes"
Sep 16 13:16:42.993658 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.993624    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"]
Sep 16 13:16:42.994124 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.994096    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78150451-e55d-4136-8c32-eff3f8158a8a" containerName="manager"
Sep 16 13:16:42.994124 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.994111    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78150451-e55d-4136-8c32-eff3f8158a8a" containerName="manager"
Sep 16 13:16:42.994212 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.994194    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78150451-e55d-4136-8c32-eff3f8158a8a" containerName="manager"
Sep 16 13:16:42.996594 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.996571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:42.998935 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.998908    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:16:42.999171 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.999154    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2pxnb\""
Sep 16 13:16:42.999226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.999210    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 13:16:42.999269 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:42.999231    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 13:16:43.010162 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.010135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"]
Sep 16 13:16:43.061450 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.061450 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.061644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.061644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.061644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.061644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3523ece8-8647-442a-bef4-8f7af4cf0b01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.061644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.061847 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.061847 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.061685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g75k\" (UniqueName: \"kubernetes.io/projected/3523ece8-8647-442a-bef4-8f7af4cf0b01-kube-api-access-2g75k\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.162793 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.162757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.162793 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.162793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.162818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.162881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.162920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.162965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3523ece8-8647-442a-bef4-8f7af4cf0b01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.163008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.163035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.163061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g75k\" (UniqueName: \"kubernetes.io/projected/3523ece8-8647-442a-bef4-8f7af4cf0b01-kube-api-access-2g75k\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.163219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163578 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.163532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163578 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.163546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163745 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.163722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3523ece8-8647-442a-bef4-8f7af4cf0b01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.163806 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.163763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.165230 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.165209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.165356 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.165339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.172076 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.172047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3523ece8-8647-442a-bef4-8f7af4cf0b01-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.172157 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.172138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g75k\" (UniqueName: \"kubernetes.io/projected/3523ece8-8647-442a-bef4-8f7af4cf0b01-kube-api-access-2g75k\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v5cfk\" (UID: \"3523ece8-8647-442a-bef4-8f7af4cf0b01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.307300 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.307214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:43.579676 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.579650    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"]
Sep 16 13:16:43.582305 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:16:43.582272    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3523ece8_8647_442a_bef4_8f7af4cf0b01.slice/crio-3b4464d40c1b5005267d2fff65d90a57dbd7e2736dbc4932c2c8925a473a4c31 WatchSource:0}: Error finding container 3b4464d40c1b5005267d2fff65d90a57dbd7e2736dbc4932c2c8925a473a4c31: Status 404 returned error can't find the container with id 3b4464d40c1b5005267d2fff65d90a57dbd7e2736dbc4932c2c8925a473a4c31
Sep 16 13:16:43.584393 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.584359    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:16:43.584473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.584424    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:16:43.584473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:43.584452    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:16:44.378369 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:44.378330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" event={"ID":"3523ece8-8647-442a-bef4-8f7af4cf0b01","Type":"ContainerStarted","Data":"54e99b7dfacac477e160939126f9b6e6ffd1093a1a0a823c7a549eaefd10d045"}
Sep 16 13:16:44.378747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:44.378387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" event={"ID":"3523ece8-8647-442a-bef4-8f7af4cf0b01","Type":"ContainerStarted","Data":"3b4464d40c1b5005267d2fff65d90a57dbd7e2736dbc4932c2c8925a473a4c31"}
Sep 16 13:16:44.400001 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:44.399952    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" podStartSLOduration=2.399938319 podStartE2EDuration="2.399938319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:44.39817775 +0000 UTC m=+475.820919342" watchObservedRunningTime="2026-09-16 13:16:44.399938319 +0000 UTC m=+475.822679901"
Sep 16 13:16:45.307921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:45.307886    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:45.312541 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:45.312518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:45.383181 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:45.383151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:45.384246 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:45.384227    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:16:47.647930 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.647897    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-b2dxw"]
Sep 16 13:16:47.650676 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.650652    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:47.652895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.652874    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 13:16:47.653005 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.652936    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hkwhd\""
Sep 16 13:16:47.662151 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.662127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-b2dxw"]
Sep 16 13:16:47.708811 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.708773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9abcc195-8e87-4687-ae3b-0b2d415027b8-config-file\") pod \"limitador-limitador-8588749977-b2dxw\" (UID: \"9abcc195-8e87-4687-ae3b-0b2d415027b8\") " pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:47.708995 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.708979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xs4\" (UniqueName: \"kubernetes.io/projected/9abcc195-8e87-4687-ae3b-0b2d415027b8-kube-api-access-56xs4\") pod \"limitador-limitador-8588749977-b2dxw\" (UID: \"9abcc195-8e87-4687-ae3b-0b2d415027b8\") " pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:47.747792 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.747755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-b2dxw"]
Sep 16 13:16:47.810129 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.810093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xs4\" (UniqueName: \"kubernetes.io/projected/9abcc195-8e87-4687-ae3b-0b2d415027b8-kube-api-access-56xs4\") pod \"limitador-limitador-8588749977-b2dxw\" (UID: \"9abcc195-8e87-4687-ae3b-0b2d415027b8\") " pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:47.810292 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.810144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9abcc195-8e87-4687-ae3b-0b2d415027b8-config-file\") pod \"limitador-limitador-8588749977-b2dxw\" (UID: \"9abcc195-8e87-4687-ae3b-0b2d415027b8\") " pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:47.810709 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.810692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9abcc195-8e87-4687-ae3b-0b2d415027b8-config-file\") pod \"limitador-limitador-8588749977-b2dxw\" (UID: \"9abcc195-8e87-4687-ae3b-0b2d415027b8\") " pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:47.819331 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.819309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xs4\" (UniqueName: \"kubernetes.io/projected/9abcc195-8e87-4687-ae3b-0b2d415027b8-kube-api-access-56xs4\") pod \"limitador-limitador-8588749977-b2dxw\" (UID: \"9abcc195-8e87-4687-ae3b-0b2d415027b8\") " pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:47.962688 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:47.962648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:48.233215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.233187    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-b2dxw"]
Sep 16 13:16:48.234912 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:16:48.234886    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abcc195_8e87_4687_ae3b_0b2d415027b8.slice/crio-b384e33272cff7c54ebc01a52514a7427f9543bf10d4f6c208215c2c3eafceaf WatchSource:0}: Error finding container b384e33272cff7c54ebc01a52514a7427f9543bf10d4f6c208215c2c3eafceaf: Status 404 returned error can't find the container with id b384e33272cff7c54ebc01a52514a7427f9543bf10d4f6c208215c2c3eafceaf
Sep 16 13:16:48.400848 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.400808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw" event={"ID":"9abcc195-8e87-4687-ae3b-0b2d415027b8","Type":"ContainerStarted","Data":"b384e33272cff7c54ebc01a52514a7427f9543bf10d4f6c208215c2c3eafceaf"}
Sep 16 13:16:48.467426 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.467398    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bfllp"]
Sep 16 13:16:48.471531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.471505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp"
Sep 16 13:16:48.473350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.473332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b6bvs\""
Sep 16 13:16:48.477385 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.477364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bfllp"]
Sep 16 13:16:48.516935 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.516840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rt7\" (UniqueName: \"kubernetes.io/projected/e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31-kube-api-access-29rt7\") pod \"authorino-7bcfd8f9cd-bfllp\" (UID: \"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp"
Sep 16 13:16:48.617803 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.617770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rt7\" (UniqueName: \"kubernetes.io/projected/e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31-kube-api-access-29rt7\") pod \"authorino-7bcfd8f9cd-bfllp\" (UID: \"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp"
Sep 16 13:16:48.631283 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.631253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rt7\" (UniqueName: \"kubernetes.io/projected/e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31-kube-api-access-29rt7\") pod \"authorino-7bcfd8f9cd-bfllp\" (UID: \"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp"
Sep 16 13:16:48.782202 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:48.782119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp"
Sep 16 13:16:49.051242 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:49.051220    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bfllp"]
Sep 16 13:16:49.053162 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:16:49.053139    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8ef1a5b_ab2b_4acf_a5cf_e0172cb08d31.slice/crio-e503f3f7aec4d44ce85d65e85b742d0885f71b5825f5a2838b6ec00ad1f755d7 WatchSource:0}: Error finding container e503f3f7aec4d44ce85d65e85b742d0885f71b5825f5a2838b6ec00ad1f755d7: Status 404 returned error can't find the container with id e503f3f7aec4d44ce85d65e85b742d0885f71b5825f5a2838b6ec00ad1f755d7
Sep 16 13:16:49.407502 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:49.407425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp" event={"ID":"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31","Type":"ContainerStarted","Data":"e503f3f7aec4d44ce85d65e85b742d0885f71b5825f5a2838b6ec00ad1f755d7"}
Sep 16 13:16:50.415326 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:50.415280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw" event={"ID":"9abcc195-8e87-4687-ae3b-0b2d415027b8","Type":"ContainerStarted","Data":"906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e"}
Sep 16 13:16:50.415730 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:50.415407    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:16:50.435443 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:50.435383    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw" podStartSLOduration=2.190218305 podStartE2EDuration="3.435363494s" totalImagesPullingTime="1.245145189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:50.433347067 +0000 UTC m=+481.856088646" watchObservedRunningTime="2026-09-16 13:16:50.435363494 +0000 UTC m=+481.858105074"
Sep 16 13:16:51.421339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:51.421305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp" event={"ID":"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31","Type":"ContainerStarted","Data":"97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648"}
Sep 16 13:16:51.441122 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:51.441074    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp" podStartSLOduration=1.832520174 podStartE2EDuration="3.441060521s" totalImagesPullingTime="1.608540347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:51.438757805 +0000 UTC m=+482.861499387" watchObservedRunningTime="2026-09-16 13:16:51.441060521 +0000 UTC m=+482.863802099"
Sep 16 13:16:51.885134 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:51.885094    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bfllp"]
Sep 16 13:16:53.431232 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:53.431192    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp" podUID="e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31" containerName="authorino" containerID="cri-o://97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648" gracePeriod=30
Sep 16 13:16:53.675436 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:53.675411    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp"
Sep 16 13:16:53.765261 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:53.765230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29rt7\" (UniqueName: \"kubernetes.io/projected/e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31-kube-api-access-29rt7\") pod \"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31\" (UID: \"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31\") "
Sep 16 13:16:53.767267 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:53.767237    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31-kube-api-access-29rt7" (OuterVolumeSpecName: "kube-api-access-29rt7") pod "e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31" (UID: "e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31"). InnerVolumeSpecName "kube-api-access-29rt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:53.866426 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:53.866373    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rt7\" (UniqueName: \"kubernetes.io/projected/e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31-kube-api-access-29rt7\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:16:54.437176 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.437144    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31" containerID="97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648" exitCode=0
Sep 16 13:16:54.437621 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.437183    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp"
Sep 16 13:16:54.437621 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.437214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp" event={"ID":"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31","Type":"ContainerDied","Data":"97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648"}
Sep 16 13:16:54.437621 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.437246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bfllp" event={"ID":"e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31","Type":"ContainerDied","Data":"e503f3f7aec4d44ce85d65e85b742d0885f71b5825f5a2838b6ec00ad1f755d7"}
Sep 16 13:16:54.437621 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.437262    2577 scope.go:117] "RemoveContainer" containerID="97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648"
Sep 16 13:16:54.445706 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.445688    2577 scope.go:117] "RemoveContainer" containerID="97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648"
Sep 16 13:16:54.446029 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:16:54.446011    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648\": container with ID starting with 97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648 not found: ID does not exist" containerID="97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648"
Sep 16 13:16:54.446085 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.446039    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648"} err="failed to get container status \"97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648\": rpc error: code = NotFound desc = could not find container \"97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648\": container with ID starting with 97afc7ba4c23eef8a62e7132efd80a53cbbc8b65cb495cc79c5a433306e6f648 not found: ID does not exist"
Sep 16 13:16:54.458229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.458208    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bfllp"]
Sep 16 13:16:54.464032 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:54.464006    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bfllp"]
Sep 16 13:16:55.109401 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:16:55.109368    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31" path="/var/lib/kubelet/pods/e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31/volumes"
Sep 16 13:17:01.422519 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:01.422483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:17:01.859499 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:01.859458    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-b2dxw"]
Sep 16 13:17:01.859701 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:01.859660    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw" podUID="9abcc195-8e87-4687-ae3b-0b2d415027b8" containerName="limitador" containerID="cri-o://906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e" gracePeriod=30
Sep 16 13:17:02.409886 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.409846    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:17:02.477133 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.477104    2577 generic.go:356] "Generic (PLEG): container finished" podID="9abcc195-8e87-4687-ae3b-0b2d415027b8" containerID="906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e" exitCode=0
Sep 16 13:17:02.477549 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.477146    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw"
Sep 16 13:17:02.477549 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.477163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw" event={"ID":"9abcc195-8e87-4687-ae3b-0b2d415027b8","Type":"ContainerDied","Data":"906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e"}
Sep 16 13:17:02.477549 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.477204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-b2dxw" event={"ID":"9abcc195-8e87-4687-ae3b-0b2d415027b8","Type":"ContainerDied","Data":"b384e33272cff7c54ebc01a52514a7427f9543bf10d4f6c208215c2c3eafceaf"}
Sep 16 13:17:02.477549 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.477226    2577 scope.go:117] "RemoveContainer" containerID="906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e"
Sep 16 13:17:02.485700 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.485682    2577 scope.go:117] "RemoveContainer" containerID="906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e"
Sep 16 13:17:02.485999 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:17:02.485979    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e\": container with ID starting with 906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e not found: ID does not exist" containerID="906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e"
Sep 16 13:17:02.486053 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.486008    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e"} err="failed to get container status \"906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e\": rpc error: code = NotFound desc = could not find container \"906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e\": container with ID starting with 906af20806c10f3e3bf2c19342b8cb4f82cb9409977aa5064cbf46966a0b072e not found: ID does not exist"
Sep 16 13:17:02.543544 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.543508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9abcc195-8e87-4687-ae3b-0b2d415027b8-config-file\") pod \"9abcc195-8e87-4687-ae3b-0b2d415027b8\" (UID: \"9abcc195-8e87-4687-ae3b-0b2d415027b8\") "
Sep 16 13:17:02.543698 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.543577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56xs4\" (UniqueName: \"kubernetes.io/projected/9abcc195-8e87-4687-ae3b-0b2d415027b8-kube-api-access-56xs4\") pod \"9abcc195-8e87-4687-ae3b-0b2d415027b8\" (UID: \"9abcc195-8e87-4687-ae3b-0b2d415027b8\") "
Sep 16 13:17:02.543984 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.543959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abcc195-8e87-4687-ae3b-0b2d415027b8-config-file" (OuterVolumeSpecName: "config-file") pod "9abcc195-8e87-4687-ae3b-0b2d415027b8" (UID: "9abcc195-8e87-4687-ae3b-0b2d415027b8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:17:02.545656 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.545623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abcc195-8e87-4687-ae3b-0b2d415027b8-kube-api-access-56xs4" (OuterVolumeSpecName: "kube-api-access-56xs4") pod "9abcc195-8e87-4687-ae3b-0b2d415027b8" (UID: "9abcc195-8e87-4687-ae3b-0b2d415027b8"). InnerVolumeSpecName "kube-api-access-56xs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:17:02.645204 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.645169    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9abcc195-8e87-4687-ae3b-0b2d415027b8-config-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:17:02.645204 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.645200    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xs4\" (UniqueName: \"kubernetes.io/projected/9abcc195-8e87-4687-ae3b-0b2d415027b8-kube-api-access-56xs4\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:17:02.801110 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.801077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-b2dxw"]
Sep 16 13:17:02.807101 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:02.807077    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-b2dxw"]
Sep 16 13:17:03.109760 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:03.109674    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abcc195-8e87-4687-ae3b-0b2d415027b8" path="/var/lib/kubelet/pods/9abcc195-8e87-4687-ae3b-0b2d415027b8/volumes"
Sep 16 13:17:07.527165 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.527130    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"]
Sep 16 13:17:07.527535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.527523    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31" containerName="authorino"
Sep 16 13:17:07.527581 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.527536    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31" containerName="authorino"
Sep 16 13:17:07.527581 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.527551    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abcc195-8e87-4687-ae3b-0b2d415027b8" containerName="limitador"
Sep 16 13:17:07.527581 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.527556    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abcc195-8e87-4687-ae3b-0b2d415027b8" containerName="limitador"
Sep 16 13:17:07.527671 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.527616    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abcc195-8e87-4687-ae3b-0b2d415027b8" containerName="limitador"
Sep 16 13:17:07.527671 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.527625    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ef1a5b-ab2b-4acf-a5cf-e0172cb08d31" containerName="authorino"
Sep 16 13:17:07.530549 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.530532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:07.536209 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.536181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 13:17:07.536209 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.536193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dqs4l\""
Sep 16 13:17:07.536924 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.536900    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:17:07.539528 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.539508    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 13:17:07.552102 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.552079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"]
Sep 16 13:17:07.692761 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.692724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae5cac71-0c53-49de-af6b-cb9dba3d1ee3-data\") pod \"postgres-69fb4594cf-k6k4h\" (UID: \"ae5cac71-0c53-49de-af6b-cb9dba3d1ee3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:07.692996 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.692845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jcn\" (UniqueName: \"kubernetes.io/projected/ae5cac71-0c53-49de-af6b-cb9dba3d1ee3-kube-api-access-n2jcn\") pod \"postgres-69fb4594cf-k6k4h\" (UID: \"ae5cac71-0c53-49de-af6b-cb9dba3d1ee3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:07.793794 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.793707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jcn\" (UniqueName: \"kubernetes.io/projected/ae5cac71-0c53-49de-af6b-cb9dba3d1ee3-kube-api-access-n2jcn\") pod \"postgres-69fb4594cf-k6k4h\" (UID: \"ae5cac71-0c53-49de-af6b-cb9dba3d1ee3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:07.793794 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.793781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae5cac71-0c53-49de-af6b-cb9dba3d1ee3-data\") pod \"postgres-69fb4594cf-k6k4h\" (UID: \"ae5cac71-0c53-49de-af6b-cb9dba3d1ee3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:07.794146 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.794131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae5cac71-0c53-49de-af6b-cb9dba3d1ee3-data\") pod \"postgres-69fb4594cf-k6k4h\" (UID: \"ae5cac71-0c53-49de-af6b-cb9dba3d1ee3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:07.806736 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.806713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jcn\" (UniqueName: \"kubernetes.io/projected/ae5cac71-0c53-49de-af6b-cb9dba3d1ee3-kube-api-access-n2jcn\") pod \"postgres-69fb4594cf-k6k4h\" (UID: \"ae5cac71-0c53-49de-af6b-cb9dba3d1ee3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:07.840493 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:07.840467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:08.132789 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:17:08.132760    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5cac71_0c53_49de_af6b_cb9dba3d1ee3.slice/crio-6065f5291ae536369d294f923883572be3bed3ea14d06d2313e687b9d54e3938 WatchSource:0}: Error finding container 6065f5291ae536369d294f923883572be3bed3ea14d06d2313e687b9d54e3938: Status 404 returned error can't find the container with id 6065f5291ae536369d294f923883572be3bed3ea14d06d2313e687b9d54e3938
Sep 16 13:17:08.134992 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:08.134963    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"]
Sep 16 13:17:08.508744 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:08.508709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h" event={"ID":"ae5cac71-0c53-49de-af6b-cb9dba3d1ee3","Type":"ContainerStarted","Data":"6065f5291ae536369d294f923883572be3bed3ea14d06d2313e687b9d54e3938"}
Sep 16 13:17:13.539442 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:13.539401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h" event={"ID":"ae5cac71-0c53-49de-af6b-cb9dba3d1ee3","Type":"ContainerStarted","Data":"d37496f88b561c5ed820499d43143c0550c08fd3d8de28a7bb24f11fe2220054"}
Sep 16 13:17:13.539976 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:13.539467    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:13.560765 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:13.560709    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h" podStartSLOduration=1.672213725 podStartE2EDuration="6.560693247s" totalImagesPullingTime="4.888479522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:17:13.558286725 +0000 UTC m=+504.981028304" watchObservedRunningTime="2026-09-16 13:17:13.560693247 +0000 UTC m=+504.983434826"
Sep 16 13:17:19.574456 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:19.574423    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k6k4h"
Sep 16 13:17:20.464499 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.464462    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hknsl"]
Sep 16 13:17:20.469419 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.469400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:20.474215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.474191    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 13:17:20.474339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.474191    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b6bvs\""
Sep 16 13:17:20.500192 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.500162    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hknsl"]
Sep 16 13:17:20.510544 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.510516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6ccfe90-2ac2-4232-985f-70c4c15b926d-tls-cert\") pod \"authorino-6d4dfc649f-hknsl\" (UID: \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\") " pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:20.510701 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.510550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bfs\" (UniqueName: \"kubernetes.io/projected/a6ccfe90-2ac2-4232-985f-70c4c15b926d-kube-api-access-z6bfs\") pod \"authorino-6d4dfc649f-hknsl\" (UID: \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\") " pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:20.611051 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.611015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6ccfe90-2ac2-4232-985f-70c4c15b926d-tls-cert\") pod \"authorino-6d4dfc649f-hknsl\" (UID: \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\") " pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:20.611507 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.611066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bfs\" (UniqueName: \"kubernetes.io/projected/a6ccfe90-2ac2-4232-985f-70c4c15b926d-kube-api-access-z6bfs\") pod \"authorino-6d4dfc649f-hknsl\" (UID: \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\") " pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:20.613528 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.613506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6ccfe90-2ac2-4232-985f-70c4c15b926d-tls-cert\") pod \"authorino-6d4dfc649f-hknsl\" (UID: \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\") " pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:20.643558 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.643524    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hknsl"]
Sep 16 13:17:20.643828 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:17:20.643809    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-z6bfs], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-hknsl" podUID="a6ccfe90-2ac2-4232-985f-70c4c15b926d"
Sep 16 13:17:20.652556 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.652527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bfs\" (UniqueName: \"kubernetes.io/projected/a6ccfe90-2ac2-4232-985f-70c4c15b926d-kube-api-access-z6bfs\") pod \"authorino-6d4dfc649f-hknsl\" (UID: \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\") " pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:20.706552 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.706517    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c45b99978-frxr8"]
Sep 16 13:17:20.710151 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.710134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:17:20.729214 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.729186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c45b99978-frxr8"]
Sep 16 13:17:20.812935 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.812897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-tls-cert\") pod \"authorino-c45b99978-frxr8\" (UID: \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\") " pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:17:20.813140 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.813043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-kube-api-access-rz9ps\") pod \"authorino-c45b99978-frxr8\" (UID: \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\") " pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:17:20.914523 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.914485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-tls-cert\") pod \"authorino-c45b99978-frxr8\" (UID: \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\") " pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:17:20.914691 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.914585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-kube-api-access-rz9ps\") pod \"authorino-c45b99978-frxr8\" (UID: \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\") " pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:17:20.916981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.916961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-tls-cert\") pod \"authorino-c45b99978-frxr8\" (UID: \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\") " pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:17:20.929687 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:20.929655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-kube-api-access-rz9ps\") pod \"authorino-c45b99978-frxr8\" (UID: \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\") " pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:17:21.019299 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.019214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:17:21.299443 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.299418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c45b99978-frxr8"]
Sep 16 13:17:21.302194 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:17:21.302170    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0122e8a_ed43_4ef4_a6c3_2c2d2fa0faea.slice/crio-b7480ff675148191b7448480bc9c5645c6cc7c918ebb22edfba88bec2c652b88 WatchSource:0}: Error finding container b7480ff675148191b7448480bc9c5645c6cc7c918ebb22edfba88bec2c652b88: Status 404 returned error can't find the container with id b7480ff675148191b7448480bc9c5645c6cc7c918ebb22edfba88bec2c652b88
Sep 16 13:17:21.585446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.585355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c45b99978-frxr8" event={"ID":"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea","Type":"ContainerStarted","Data":"b7480ff675148191b7448480bc9c5645c6cc7c918ebb22edfba88bec2c652b88"}
Sep 16 13:17:21.585446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.585415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:21.594044 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.594023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:21.620591 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.620561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6bfs\" (UniqueName: \"kubernetes.io/projected/a6ccfe90-2ac2-4232-985f-70c4c15b926d-kube-api-access-z6bfs\") pod \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\" (UID: \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\") "
Sep 16 13:17:21.620973 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.620635    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6ccfe90-2ac2-4232-985f-70c4c15b926d-tls-cert\") pod \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\" (UID: \"a6ccfe90-2ac2-4232-985f-70c4c15b926d\") "
Sep 16 13:17:21.622679 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.622650    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ccfe90-2ac2-4232-985f-70c4c15b926d-kube-api-access-z6bfs" (OuterVolumeSpecName: "kube-api-access-z6bfs") pod "a6ccfe90-2ac2-4232-985f-70c4c15b926d" (UID: "a6ccfe90-2ac2-4232-985f-70c4c15b926d"). InnerVolumeSpecName "kube-api-access-z6bfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:17:21.622792 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.622710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ccfe90-2ac2-4232-985f-70c4c15b926d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a6ccfe90-2ac2-4232-985f-70c4c15b926d" (UID: "a6ccfe90-2ac2-4232-985f-70c4c15b926d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:17:21.722213 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.722178    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bfs\" (UniqueName: \"kubernetes.io/projected/a6ccfe90-2ac2-4232-985f-70c4c15b926d-kube-api-access-z6bfs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:17:21.722213 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:21.722206    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6ccfe90-2ac2-4232-985f-70c4c15b926d-tls-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:17:22.591473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:22.591436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c45b99978-frxr8" event={"ID":"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea","Type":"ContainerStarted","Data":"7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6"}
Sep 16 13:17:22.591473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:22.591456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hknsl"
Sep 16 13:17:22.622088 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:22.622040    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c45b99978-frxr8" podStartSLOduration=1.9049506040000002 podStartE2EDuration="2.62202365s" totalImagesPullingTime="717.073046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:17:22.621664557 +0000 UTC m=+514.044406135" watchObservedRunningTime="2026-09-16 13:17:22.62202365 +0000 UTC m=+514.044765229"
Sep 16 13:17:22.695407 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:22.695369    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hknsl"]
Sep 16 13:17:22.704273 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:22.704242    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hknsl"]
Sep 16 13:17:23.109768 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:23.109727    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6ccfe90-2ac2-4232-985f-70c4c15b926d" path="/var/lib/kubelet/pods/a6ccfe90-2ac2-4232-985f-70c4c15b926d/volumes"
Sep 16 13:17:28.486471 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.486433    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65b547448-r9hrp"]
Sep 16 13:17:28.504654 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.504621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.507521 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.507493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b547448-r9hrp"]
Sep 16 13:17:28.507639 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.507568    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 13:17:28.508290 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.508265    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 13:17:28.508411 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.508350    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 13:17:28.508411 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.508371    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hqshj\""
Sep 16 13:17:28.593134 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.593049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-metrics-certs\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.593279 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.593149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-webhook-cert\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.593279 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.593200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrwb\" (UniqueName: \"kubernetes.io/projected/97723da7-1f39-49e9-9e45-3a737e8a5b7d-kube-api-access-skrwb\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.693778 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.693741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-webhook-cert\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.693971 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.693794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrwb\" (UniqueName: \"kubernetes.io/projected/97723da7-1f39-49e9-9e45-3a737e8a5b7d-kube-api-access-skrwb\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.693971 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.693882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-metrics-certs\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.696436 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.696410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-webhook-cert\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.696436 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.696427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-metrics-certs\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.725878 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.725826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrwb\" (UniqueName: \"kubernetes.io/projected/97723da7-1f39-49e9-9e45-3a737e8a5b7d-kube-api-access-skrwb\") pod \"maas-controller-65b547448-r9hrp\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") " pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:28.817357 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:28.817324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:29.110625 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:29.110591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b547448-r9hrp"]
Sep 16 13:17:29.113463 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:17:29.113431    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97723da7_1f39_49e9_9e45_3a737e8a5b7d.slice/crio-5e6f7ab94b08baacfa2021245fe5e556fabd7ac3dd3c7264794bbca1766592a9 WatchSource:0}: Error finding container 5e6f7ab94b08baacfa2021245fe5e556fabd7ac3dd3c7264794bbca1766592a9: Status 404 returned error can't find the container with id 5e6f7ab94b08baacfa2021245fe5e556fabd7ac3dd3c7264794bbca1766592a9
Sep 16 13:17:29.624517 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:29.624486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b547448-r9hrp" event={"ID":"97723da7-1f39-49e9-9e45-3a737e8a5b7d","Type":"ContainerStarted","Data":"5e6f7ab94b08baacfa2021245fe5e556fabd7ac3dd3c7264794bbca1766592a9"}
Sep 16 13:17:31.637921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:31.637821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b547448-r9hrp" event={"ID":"97723da7-1f39-49e9-9e45-3a737e8a5b7d","Type":"ContainerStarted","Data":"8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99"}
Sep 16 13:17:31.638348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:31.638019    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:17:31.675354 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:31.675301    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65b547448-r9hrp" podStartSLOduration=1.36214744 podStartE2EDuration="3.675283957s" totalImagesPullingTime="2.313136517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:17:31.673090885 +0000 UTC m=+523.095832464" watchObservedRunningTime="2026-09-16 13:17:31.675283957 +0000 UTC m=+523.098025535"
Sep 16 13:17:42.645441 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:17:42.645359    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:18:06.019593 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.019553    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84db685c48-z98rf"]
Sep 16 13:18:06.025794 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.025770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.029050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.029025    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tmb4l\""
Sep 16 13:18:06.029050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.029044    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 13:18:06.029237 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.029045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 13:18:06.053195 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.053159    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84db685c48-z98rf"]
Sep 16 13:18:06.148803 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.148768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5296ab2-975e-464b-ad17-1428fb2fafde-openshift-service-ca\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.148803 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.148804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-metrics-certs\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.149090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.148831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjwb\" (UniqueName: \"kubernetes.io/projected/c5296ab2-975e-464b-ad17-1428fb2fafde-kube-api-access-8bjwb\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.149090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.148880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-maas-api-tls\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.250427 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.250384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5296ab2-975e-464b-ad17-1428fb2fafde-openshift-service-ca\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.250427 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.250428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-metrics-certs\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.250681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.250461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjwb\" (UniqueName: \"kubernetes.io/projected/c5296ab2-975e-464b-ad17-1428fb2fafde-kube-api-access-8bjwb\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.250681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.250489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-maas-api-tls\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.250681 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:18:06.250610    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 16 13:18:06.250839 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:18:06.250692    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-maas-api-tls podName:c5296ab2-975e-464b-ad17-1428fb2fafde nodeName:}" failed. No retries permitted until 2026-09-16 13:18:06.750669634 +0000 UTC m=+558.173411211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-maas-api-tls") pod "maas-api-84db685c48-z98rf" (UID: "c5296ab2-975e-464b-ad17-1428fb2fafde") : secret "maas-api-serving-cert" not found
Sep 16 13:18:06.251185 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.251164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5296ab2-975e-464b-ad17-1428fb2fafde-openshift-service-ca\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.252882 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.252846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-metrics-certs\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.264310 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.264290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjwb\" (UniqueName: \"kubernetes.io/projected/c5296ab2-975e-464b-ad17-1428fb2fafde-kube-api-access-8bjwb\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.754946 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.754912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-maas-api-tls\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.757466 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.757442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5296ab2-975e-464b-ad17-1428fb2fafde-maas-api-tls\") pod \"maas-api-84db685c48-z98rf\" (UID: \"c5296ab2-975e-464b-ad17-1428fb2fafde\") " pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:06.936314 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:06.936267    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:07.232823 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:07.231025    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84db685c48-z98rf"]
Sep 16 13:18:07.235530 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:18:07.235477    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5296ab2_975e_464b_ad17_1428fb2fafde.slice/crio-8299ca1757326d7c23d8ec367bcabc71aff3d4daa36dfa23c3d382fd0c031ebc WatchSource:0}: Error finding container 8299ca1757326d7c23d8ec367bcabc71aff3d4daa36dfa23c3d382fd0c031ebc: Status 404 returned error can't find the container with id 8299ca1757326d7c23d8ec367bcabc71aff3d4daa36dfa23c3d382fd0c031ebc
Sep 16 13:18:07.815566 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:07.815516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf" event={"ID":"c5296ab2-975e-464b-ad17-1428fb2fafde","Type":"ContainerStarted","Data":"8299ca1757326d7c23d8ec367bcabc71aff3d4daa36dfa23c3d382fd0c031ebc"}
Sep 16 13:18:09.827991 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:09.827947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf" event={"ID":"c5296ab2-975e-464b-ad17-1428fb2fafde","Type":"ContainerStarted","Data":"68f88ec5996fce904e4d422459c0776592d111eb1d01414424bc509f91ca0705"}
Sep 16 13:18:09.828391 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:09.828108    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:09.851363 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:09.851290    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf" podStartSLOduration=3.113130972 podStartE2EDuration="4.851268844s" totalImagesPullingTime="1.738137872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:18:09.850213706 +0000 UTC m=+561.272955282" watchObservedRunningTime="2026-09-16 13:18:09.851268844 +0000 UTC m=+561.274010423"
Sep 16 13:18:15.764428 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.764235    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v5cfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": read tcp 10.134.0.2:52428->10.134.0.32:15021: read: connection reset by peer" start-of-body=
Sep 16 13:18:15.764428 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.764324    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" podUID="3523ece8-8647-442a-bef4-8f7af4cf0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": read tcp 10.134.0.2:52428->10.134.0.32:15021: read: connection reset by peer"
Sep 16 13:18:15.842112 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.842082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84db685c48-z98rf"
Sep 16 13:18:15.866947 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.866906    2577 generic.go:356] "Generic (PLEG): container finished" podID="3523ece8-8647-442a-bef4-8f7af4cf0b01" containerID="54e99b7dfacac477e160939126f9b6e6ffd1093a1a0a823c7a549eaefd10d045" exitCode=0
Sep 16 13:18:15.867142 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.866975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" event={"ID":"3523ece8-8647-442a-bef4-8f7af4cf0b01","Type":"ContainerDied","Data":"54e99b7dfacac477e160939126f9b6e6ffd1093a1a0a823c7a549eaefd10d045"}
Sep 16 13:18:15.867448 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.867427    2577 scope.go:117] "RemoveContainer" containerID="54e99b7dfacac477e160939126f9b6e6ffd1093a1a0a823c7a549eaefd10d045"
Sep 16 13:18:15.868689 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.868663    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:18:15.868780 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.868734    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:18:15.868780 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:15.868768    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:18:16.873921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:16.873887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" event={"ID":"3523ece8-8647-442a-bef4-8f7af4cf0b01","Type":"ContainerStarted","Data":"b512d76c3ad77b72421120b22d52f35d9ca716f666239ae0fcef6114c4d5f4a3"}
Sep 16 13:18:17.308116 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:17.308081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:18:17.308328 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:17.308302    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v5cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:18:17.308376 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:17.308356    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" podUID="3523ece8-8647-442a-bef4-8f7af4cf0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 13:18:18.308446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:18.308408    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v5cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:18:18.308826 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:18.308469    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" podUID="3523ece8-8647-442a-bef4-8f7af4cf0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 13:18:19.307778 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:19.307739    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v5cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:18:19.307979 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:19.307797    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" podUID="3523ece8-8647-442a-bef4-8f7af4cf0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 13:18:20.307953 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:20.307916    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v5cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:18:20.308339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:20.307983    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" podUID="3523ece8-8647-442a-bef4-8f7af4cf0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 13:18:21.308445 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:21.308400    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v5cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:18:21.308955 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:21.308470    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk" podUID="3523ece8-8647-442a-bef4-8f7af4cf0b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 13:18:22.311386 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:22.311355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:18:22.311814 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:22.311621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:18:22.312446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:22.312424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v5cfk"
Sep 16 13:18:28.402618 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.402586    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"]
Sep 16 13:18:28.406067 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.406048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.408982 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.408959    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 13:18:28.408982 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.408970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bdc7l\""
Sep 16 13:18:28.409173 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.408988    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 13:18:28.409173 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.409029    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 13:18:28.420223 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.420194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"]
Sep 16 13:18:28.562038 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.561934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79df9923-d1de-48f3-84f4-f745d3239401-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.562038 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.561996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrzk\" (UniqueName: \"kubernetes.io/projected/79df9923-d1de-48f3-84f4-f745d3239401-kube-api-access-rrrzk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.562245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.562078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.562245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.562113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.562245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.562178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.562245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.562205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663042 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79df9923-d1de-48f3-84f4-f745d3239401-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrzk\" (UniqueName: \"kubernetes.io/projected/79df9923-d1de-48f3-84f4-f745d3239401-kube-api-access-rrrzk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663503 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663503 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663608 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.663608 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.663588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.665505 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.665483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79df9923-d1de-48f3-84f4-f745d3239401-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.665770 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.665745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79df9923-d1de-48f3-84f4-f745d3239401-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.672211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.672192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrzk\" (UniqueName: \"kubernetes.io/projected/79df9923-d1de-48f3-84f4-f745d3239401-kube-api-access-rrrzk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9\" (UID: \"79df9923-d1de-48f3-84f4-f745d3239401\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.716893 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.716847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:28.997410 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:28.997373    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"]
Sep 16 13:18:28.999833 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:18:28.999804    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79df9923_d1de_48f3_84f4_f745d3239401.slice/crio-3fdd53f8b48dc202374563bc6bf78684017d08a392b1cbbe3ba4fe5f6f506e22 WatchSource:0}: Error finding container 3fdd53f8b48dc202374563bc6bf78684017d08a392b1cbbe3ba4fe5f6f506e22: Status 404 returned error can't find the container with id 3fdd53f8b48dc202374563bc6bf78684017d08a392b1cbbe3ba4fe5f6f506e22
Sep 16 13:18:29.943648 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:29.943607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9" event={"ID":"79df9923-d1de-48f3-84f4-f745d3239401","Type":"ContainerStarted","Data":"3fdd53f8b48dc202374563bc6bf78684017d08a392b1cbbe3ba4fe5f6f506e22"}
Sep 16 13:18:30.949929 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:30.949895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9" event={"ID":"79df9923-d1de-48f3-84f4-f745d3239401","Type":"ContainerStarted","Data":"27b51e71c8db98de42e600aa4bbfc538d12055f74c941761a5a3da1ee3a078df"}
Sep 16 13:18:30.990211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:30.990156    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9" podStartSLOduration=1.8893668350000001 podStartE2EDuration="2.990139545s" totalImagesPullingTime="1.10077271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:18:30.98848029 +0000 UTC m=+582.411221893" watchObservedRunningTime="2026-09-16 13:18:30.990139545 +0000 UTC m=+582.412881124"
Sep 16 13:18:38.717206 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:38.717164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:38.717764 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:38.717276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:38.730132 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:38.730107    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:39.000578 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:39.000501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9"
Sep 16 13:18:48.003435 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.003400    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"]
Sep 16 13:18:48.006235 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.006217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.008966 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.008946    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 13:18:48.029424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.029398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"]
Sep 16 13:18:48.042255 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.042227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.042391 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.042306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.042391 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.042349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9861481-1ddf-4255-ab62-cee33f299986-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.042531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.042390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jtr\" (UniqueName: \"kubernetes.io/projected/c9861481-1ddf-4255-ab62-cee33f299986-kube-api-access-k2jtr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.042531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.042415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.042531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.042496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143517 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9861481-1ddf-4255-ab62-cee33f299986-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143517 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jtr\" (UniqueName: \"kubernetes.io/projected/c9861481-1ddf-4255-ab62-cee33f299986-kube-api-access-k2jtr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143517 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143517 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143733 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143802 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143881 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.143934 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.143887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.145676 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.145656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9861481-1ddf-4255-ab62-cee33f299986-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.146096 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.146076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9861481-1ddf-4255-ab62-cee33f299986-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.152464 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.152433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jtr\" (UniqueName: \"kubernetes.io/projected/c9861481-1ddf-4255-ab62-cee33f299986-kube-api-access-k2jtr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nk829\" (UID: \"c9861481-1ddf-4255-ab62-cee33f299986\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.316368 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.316272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:48.591323 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:48.591299    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"]
Sep 16 13:18:48.592601 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:18:48.592574    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9861481_1ddf_4255_ab62_cee33f299986.slice/crio-9268ba5d35744f170b0cddccd9f8e1c1ecfbf0b7600957c8e3dcd83f29fdcd9d WatchSource:0}: Error finding container 9268ba5d35744f170b0cddccd9f8e1c1ecfbf0b7600957c8e3dcd83f29fdcd9d: Status 404 returned error can't find the container with id 9268ba5d35744f170b0cddccd9f8e1c1ecfbf0b7600957c8e3dcd83f29fdcd9d
Sep 16 13:18:49.047468 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:49.047431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829" event={"ID":"c9861481-1ddf-4255-ab62-cee33f299986","Type":"ContainerStarted","Data":"b6e2c794a0194144b00ceaa6f2d3a488da431add8907d7436550b2f1aab24e0f"}
Sep 16 13:18:49.047867 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:49.047516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829" event={"ID":"c9861481-1ddf-4255-ab62-cee33f299986","Type":"ContainerStarted","Data":"9268ba5d35744f170b0cddccd9f8e1c1ecfbf0b7600957c8e3dcd83f29fdcd9d"}
Sep 16 13:18:49.070665 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:49.070608    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829" podStartSLOduration=1.8768323869999999 podStartE2EDuration="2.070588556s" totalImagesPullingTime="193.756169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:18:49.068620009 +0000 UTC m=+600.491361588" watchObservedRunningTime="2026-09-16 13:18:49.070588556 +0000 UTC m=+600.493330136"
Sep 16 13:18:58.316980 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:58.316933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:58.317415 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:58.317007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:58.329320 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:58.329294    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:18:59.124210 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:18:59.124178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nk829"
Sep 16 13:19:05.812615 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.812581    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"]
Sep 16 13:19:05.816327 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.816305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:05.818694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.818668    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 13:19:05.826238 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.826213    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"]
Sep 16 13:19:05.916378 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.916347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7fx\" (UniqueName: \"kubernetes.io/projected/605822f9-d69f-4669-964a-28d0ac652bed-kube-api-access-fg7fx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:05.916546 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.916414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:05.916546 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.916451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:05.916546 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.916467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605822f9-d69f-4669-964a-28d0ac652bed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:05.916546 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.916505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:05.916546 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:05.916542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.017624 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.017580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.017624 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.017625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.017893 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.017644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605822f9-d69f-4669-964a-28d0ac652bed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.017893 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.017694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.017893 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.017748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.017893 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.017814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7fx\" (UniqueName: \"kubernetes.io/projected/605822f9-d69f-4669-964a-28d0ac652bed-kube-api-access-fg7fx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.018115 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.018035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.018115 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.018094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.018229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.018207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.019997 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.019971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605822f9-d69f-4669-964a-28d0ac652bed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.020309 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.020293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605822f9-d69f-4669-964a-28d0ac652bed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.025695 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.025669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7fx\" (UniqueName: \"kubernetes.io/projected/605822f9-d69f-4669-964a-28d0ac652bed-kube-api-access-fg7fx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f\" (UID: \"605822f9-d69f-4669-964a-28d0ac652bed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.128421 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.128338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:06.402669 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.402639    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"]
Sep 16 13:19:06.405060 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:19:06.405031    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605822f9_d69f_4669_964a_28d0ac652bed.slice/crio-d3cf167f1e9cc7bad625f45aed1ea647be3d080c578e0801eb75ae857e64ceff WatchSource:0}: Error finding container d3cf167f1e9cc7bad625f45aed1ea647be3d080c578e0801eb75ae857e64ceff: Status 404 returned error can't find the container with id d3cf167f1e9cc7bad625f45aed1ea647be3d080c578e0801eb75ae857e64ceff
Sep 16 13:19:06.406500 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:06.406478    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:19:07.148056 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:07.148020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f" event={"ID":"605822f9-d69f-4669-964a-28d0ac652bed","Type":"ContainerStarted","Data":"5c6ad399937663a8a6b69fe25052bf0f502563bf8ca1402b3a6ddec9ac24729d"}
Sep 16 13:19:07.148449 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:07.148075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f" event={"ID":"605822f9-d69f-4669-964a-28d0ac652bed","Type":"ContainerStarted","Data":"d3cf167f1e9cc7bad625f45aed1ea647be3d080c578e0801eb75ae857e64ceff"}
Sep 16 13:19:07.164695 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:07.164619    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f" podStartSLOduration=1.9913838190000002 podStartE2EDuration="2.164603477s" totalImagesPullingTime="173.219658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:19:07.163038155 +0000 UTC m=+618.585779734" watchObservedRunningTime="2026-09-16 13:19:07.164603477 +0000 UTC m=+618.587345068"
Sep 16 13:19:16.128902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:16.128797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:16.128902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:16.128875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:16.141613 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:16.141586    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:19:16.204120 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:19:16.204089    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f"
Sep 16 13:20:35.801974 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:35.801936    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c45b99978-frxr8"]
Sep 16 13:20:35.803184 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:35.803145    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c45b99978-frxr8" podUID="b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea" containerName="authorino" containerID="cri-o://7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6" gracePeriod=30
Sep 16 13:20:36.056629 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.056566    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:20:36.210070 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.210041    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-tls-cert\") pod \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\" (UID: \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\") "
Sep 16 13:20:36.210070 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.210074    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-kube-api-access-rz9ps\") pod \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\" (UID: \"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea\") "
Sep 16 13:20:36.212115 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.212081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-kube-api-access-rz9ps" (OuterVolumeSpecName: "kube-api-access-rz9ps") pod "b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea" (UID: "b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea"). InnerVolumeSpecName "kube-api-access-rz9ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:20:36.220546 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.220523    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea" (UID: "b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:20:36.311837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.311741    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-tls-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:20:36.311837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.311775    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9ps\" (UniqueName: \"kubernetes.io/projected/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea-kube-api-access-rz9ps\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:20:36.609059 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.608977    2577 generic.go:356] "Generic (PLEG): container finished" podID="b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea" containerID="7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6" exitCode=0
Sep 16 13:20:36.609059 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.609019    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c45b99978-frxr8"
Sep 16 13:20:36.609059 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.609038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c45b99978-frxr8" event={"ID":"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea","Type":"ContainerDied","Data":"7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6"}
Sep 16 13:20:36.609278 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.609069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c45b99978-frxr8" event={"ID":"b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea","Type":"ContainerDied","Data":"b7480ff675148191b7448480bc9c5645c6cc7c918ebb22edfba88bec2c652b88"}
Sep 16 13:20:36.609278 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.609085    2577 scope.go:117] "RemoveContainer" containerID="7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6"
Sep 16 13:20:36.620941 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.620918    2577 scope.go:117] "RemoveContainer" containerID="7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6"
Sep 16 13:20:36.621352 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:20:36.621333    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6\": container with ID starting with 7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6 not found: ID does not exist" containerID="7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6"
Sep 16 13:20:36.621410 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.621361    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6"} err="failed to get container status \"7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6\": rpc error: code = NotFound desc = could not find container \"7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6\": container with ID starting with 7c8ac5c55843de120bab6588ca0be0b1686da7740168a0611283f170c40c5dc6 not found: ID does not exist"
Sep 16 13:20:36.629563 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.629535    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c45b99978-frxr8"]
Sep 16 13:20:36.633491 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:36.633463    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c45b99978-frxr8"]
Sep 16 13:20:37.110188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:37.110145    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea" path="/var/lib/kubelet/pods/b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea/volumes"
Sep 16 13:20:59.349036 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.349000    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"]
Sep 16 13:20:59.349520 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.349408    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea" containerName="authorino"
Sep 16 13:20:59.349520 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.349420    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea" containerName="authorino"
Sep 16 13:20:59.349520 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.349499    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0122e8a-ed43-4ef4-a6c3-2c2d2fa0faea" containerName="authorino"
Sep 16 13:20:59.352427 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.352406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.355829 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.355808    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f27ea8-openshift-default-dockercfg-nxr6b\""
Sep 16 13:20:59.371899 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.371835    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"]
Sep 16 13:20:59.440505 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istiod-ca-cert\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.440689 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-token\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.440689 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-certs\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.440689 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-socket\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.440689 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-data\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.440940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-envoy\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.440940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-podinfo\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.440940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmh2\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-kube-api-access-8jmh2\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.440940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.440894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-credential-socket\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.534133 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.534096    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"]
Sep 16 13:20:59.537154 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.537130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.539544 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.539523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-48adf5-openshift-default-dockercfg-hpv9m\""
Sep 16 13:20:59.541589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istiod-ca-cert\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.541711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-token\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.541711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-certs\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.541711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-socket\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.541711 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-data\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.542016 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-envoy\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.542016 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-podinfo\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.542016 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmh2\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-kube-api-access-8jmh2\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.542016 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.541943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-credential-socket\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.542432 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.542385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istiod-ca-cert\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.543124 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.543098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-credential-socket\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.543318 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.543283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-socket\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.543411 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.543314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-certs\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.543518 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.543494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-data\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.544903 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.544883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-envoy\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.545562 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.545536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-podinfo\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.548194 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.548129    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"]
Sep 16 13:20:59.553648 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.553616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-token\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.553757 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.553727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmh2\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-kube-api-access-8jmh2\") pod \"e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") " pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.642674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-podinfo\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.642674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istiod-ca-cert\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.642674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-data\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.643012 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-token\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.643012 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64x4m\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-kube-api-access-64x4m\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.643012 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-certs\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.643012 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-socket\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.643012 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-credential-socket\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.643012 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.642965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-envoy\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.663038 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.663003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:20:59.743832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.743794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-certs\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.743886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-socket\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.743927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-credential-socket\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.743949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-envoy\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-podinfo\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744281 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istiod-ca-cert\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744281 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-data\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744281 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-certs\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744281 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-token\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744281 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64x4m\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-kube-api-access-64x4m\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744281 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-credential-socket\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-socket\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.744606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.744406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-data\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.745169 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.745145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istiod-ca-cert\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.747728 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.747622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-podinfo\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.747939 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.747913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-envoy\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.752433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.752411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64x4m\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-kube-api-access-64x4m\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.752953 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.752930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-token\") pod \"e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") " pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.873718 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.873685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:20:59.950640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.950603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"]
Sep 16 13:20:59.956040 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:20:59.955980    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf29a3b_e178_4d37_9a2e_421e8c8d00c9.slice/crio-9635c0b8ee6f395333b5e82c470184ef166229984c31f8e60954c9d4005ba477 WatchSource:0}: Error finding container 9635c0b8ee6f395333b5e82c470184ef166229984c31f8e60954c9d4005ba477: Status 404 returned error can't find the container with id 9635c0b8ee6f395333b5e82c470184ef166229984c31f8e60954c9d4005ba477
Sep 16 13:20:59.958379 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.958340    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:20:59.958481 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.958439    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:20:59.958534 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:20:59.958478    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:00.152515 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.152485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"]
Sep 16 13:21:00.155973 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:21:00.155930    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5b0ea4_eb39_493b_8c35_1a9e3918726b.slice/crio-2afea5d0cd9b64d6473a628229c85d3b80d668c54797b8020844f9c584a0295d WatchSource:0}: Error finding container 2afea5d0cd9b64d6473a628229c85d3b80d668c54797b8020844f9c584a0295d: Status 404 returned error can't find the container with id 2afea5d0cd9b64d6473a628229c85d3b80d668c54797b8020844f9c584a0295d
Sep 16 13:21:00.158514 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.158471    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:00.158616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.158556    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:00.158616 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.158600    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:00.736496 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.736461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" event={"ID":"ab5b0ea4-eb39-493b-8c35-1a9e3918726b","Type":"ContainerStarted","Data":"f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183"}
Sep 16 13:21:00.736990 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.736530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" event={"ID":"ab5b0ea4-eb39-493b-8c35-1a9e3918726b","Type":"ContainerStarted","Data":"2afea5d0cd9b64d6473a628229c85d3b80d668c54797b8020844f9c584a0295d"}
Sep 16 13:21:00.738338 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.738315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" event={"ID":"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9","Type":"ContainerStarted","Data":"1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba"}
Sep 16 13:21:00.738417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.738358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" event={"ID":"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9","Type":"ContainerStarted","Data":"9635c0b8ee6f395333b5e82c470184ef166229984c31f8e60954c9d4005ba477"}
Sep 16 13:21:00.763192 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.763139    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" podStartSLOduration=1.763121855 podStartE2EDuration="1.763121855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:00.761831138 +0000 UTC m=+732.184572716" watchObservedRunningTime="2026-09-16 13:21:00.763121855 +0000 UTC m=+732.185863515"
Sep 16 13:21:00.793970 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.793920    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" podStartSLOduration=1.793902898 podStartE2EDuration="1.793902898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:00.792089792 +0000 UTC m=+732.214831371" watchObservedRunningTime="2026-09-16 13:21:00.793902898 +0000 UTC m=+732.216644477"
Sep 16 13:21:00.874843 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:00.874803    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:21:01.663701 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:01.663664    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:21:01.668470 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:01.668439    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:21:01.744443 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:01.744408    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:21:01.745405 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:01.745386    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:21:01.879575 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:01.879545    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:21:02.749950 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:02.749914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:21:02.750789 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:02.750771    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:21:04.526976 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.526943    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"]
Sep 16 13:21:04.530741 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.530713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.532990 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.532967    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f27ea8-serving-cert\""
Sep 16 13:21:04.541088 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.541061    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"]
Sep 16 13:21:04.599188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.599147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-metrics-certs\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.599343 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.599206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/291680aa-20e5-4d60-879a-65bfdfa71205-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.599343 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.599272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.599498 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.599336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jtdn\" (UniqueName: \"kubernetes.io/projected/291680aa-20e5-4d60-879a-65bfdfa71205-kube-api-access-6jtdn\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.700654 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.700614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jtdn\" (UniqueName: \"kubernetes.io/projected/291680aa-20e5-4d60-879a-65bfdfa71205-kube-api-access-6jtdn\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.700832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.700710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-metrics-certs\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.700832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.700743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/291680aa-20e5-4d60-879a-65bfdfa71205-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.700832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.700786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.701070 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:21:04.700924    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-serving-cert: secret "maas-api-e2e-worker-w4-f27ea8-serving-cert" not found
Sep 16 13:21:04.701070 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:21:04.700994    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-maas-api-tls podName:291680aa-20e5-4d60-879a-65bfdfa71205 nodeName:}" failed. No retries permitted until 2026-09-16 13:21:05.200976136 +0000 UTC m=+736.623717693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-maas-api-tls") pod "maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7" (UID: "291680aa-20e5-4d60-879a-65bfdfa71205") : secret "maas-api-e2e-worker-w4-f27ea8-serving-cert" not found
Sep 16 13:21:04.701472 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.701453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/291680aa-20e5-4d60-879a-65bfdfa71205-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.703221 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.703205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-metrics-certs\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:04.710985 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:04.710963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jtdn\" (UniqueName: \"kubernetes.io/projected/291680aa-20e5-4d60-879a-65bfdfa71205-kube-api-access-6jtdn\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:05.206444 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.206406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:05.208772 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.208750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291680aa-20e5-4d60-879a-65bfdfa71205-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7\" (UID: \"291680aa-20e5-4d60-879a-65bfdfa71205\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:05.445830 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.445779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:05.589173 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.589142    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"]
Sep 16 13:21:05.593411 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.593383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.595486 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.595461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-48adf5-serving-cert\""
Sep 16 13:21:05.606792 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.606688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"]
Sep 16 13:21:05.712051 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.711969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-metrics-certs\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.712051 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.712021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vtq\" (UniqueName: \"kubernetes.io/projected/1a80e599-65c3-4d9a-bdb2-871965459827-kube-api-access-n6vtq\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.712051 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.712048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a80e599-65c3-4d9a-bdb2-871965459827-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.712323 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.712119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-maas-api-tls\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.727072 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.727046    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"]
Sep 16 13:21:05.728663 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:21:05.728640    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291680aa_20e5_4d60_879a_65bfdfa71205.slice/crio-15dcf029a13f5fd389aeaa48b02fbfa206ab5024635c62a5cc2abf7435c0f6ed WatchSource:0}: Error finding container 15dcf029a13f5fd389aeaa48b02fbfa206ab5024635c62a5cc2abf7435c0f6ed: Status 404 returned error can't find the container with id 15dcf029a13f5fd389aeaa48b02fbfa206ab5024635c62a5cc2abf7435c0f6ed
Sep 16 13:21:05.767694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.767664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7" event={"ID":"291680aa-20e5-4d60-879a-65bfdfa71205","Type":"ContainerStarted","Data":"15dcf029a13f5fd389aeaa48b02fbfa206ab5024635c62a5cc2abf7435c0f6ed"}
Sep 16 13:21:05.813207 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.813175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-metrics-certs\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.813372 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.813235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vtq\" (UniqueName: \"kubernetes.io/projected/1a80e599-65c3-4d9a-bdb2-871965459827-kube-api-access-n6vtq\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.813372 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.813266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a80e599-65c3-4d9a-bdb2-871965459827-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.813372 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.813303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-maas-api-tls\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.813530 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:21:05.813492    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-serving-cert: secret "maas-api-e2e-inf-a-48adf5-serving-cert" not found
Sep 16 13:21:05.813573 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:21:05.813561    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-maas-api-tls podName:1a80e599-65c3-4d9a-bdb2-871965459827 nodeName:}" failed. No retries permitted until 2026-09-16 13:21:06.313542693 +0000 UTC m=+737.736284257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-maas-api-tls") pod "maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq" (UID: "1a80e599-65c3-4d9a-bdb2-871965459827") : secret "maas-api-e2e-inf-a-48adf5-serving-cert" not found
Sep 16 13:21:05.814228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.814210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a80e599-65c3-4d9a-bdb2-871965459827-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.815785 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.815763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-metrics-certs\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:05.823808 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:05.823788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vtq\" (UniqueName: \"kubernetes.io/projected/1a80e599-65c3-4d9a-bdb2-871965459827-kube-api-access-n6vtq\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:06.318581 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.318548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-maas-api-tls\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:06.321021 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.320990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a80e599-65c3-4d9a-bdb2-871965459827-maas-api-tls\") pod \"maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq\" (UID: \"1a80e599-65c3-4d9a-bdb2-871965459827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:06.509666 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.509582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:06.686283 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.686247    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"]
Sep 16 13:21:06.697943 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.697908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.700103 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.700076    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a21a40-serving-cert\""
Sep 16 13:21:06.706041 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.706013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"]
Sep 16 13:21:06.778325 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.778245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7" event={"ID":"291680aa-20e5-4d60-879a-65bfdfa71205","Type":"ContainerStarted","Data":"d8eb1450a5cc1c81c77d3ada1f9c14ceb52eaeae74e020e5e16c36e5533452a8"}
Sep 16 13:21:06.778473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.778376    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:06.787625 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.787600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"]
Sep 16 13:21:06.789821 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:21:06.789793    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a80e599_65c3_4d9a_bdb2_871965459827.slice/crio-dab2de9ae76eb090a79e657c4e4308ef5cc6ad4f802b54cf952d437f7ea8fa08 WatchSource:0}: Error finding container dab2de9ae76eb090a79e657c4e4308ef5cc6ad4f802b54cf952d437f7ea8fa08: Status 404 returned error can't find the container with id dab2de9ae76eb090a79e657c4e4308ef5cc6ad4f802b54cf952d437f7ea8fa08
Sep 16 13:21:06.812410 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.812359    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7" podStartSLOduration=2.303884074 podStartE2EDuration="2.812340107s" totalImagesPullingTime="508.456033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:06.811142197 +0000 UTC m=+738.233883776" watchObservedRunningTime="2026-09-16 13:21:06.812340107 +0000 UTC m=+738.235081677"
Sep 16 13:21:06.823611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.823580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428df5de-7c4b-47ce-a02b-ff2e33a01217-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.823724 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.823636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428df5de-7c4b-47ce-a02b-ff2e33a01217-metrics-certs\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.823724 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.823693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd467\" (UniqueName: \"kubernetes.io/projected/428df5de-7c4b-47ce-a02b-ff2e33a01217-kube-api-access-vd467\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.823905 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.823745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428df5de-7c4b-47ce-a02b-ff2e33a01217-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.925481 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.925436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428df5de-7c4b-47ce-a02b-ff2e33a01217-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.925677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.925531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428df5de-7c4b-47ce-a02b-ff2e33a01217-metrics-certs\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.925677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.925568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd467\" (UniqueName: \"kubernetes.io/projected/428df5de-7c4b-47ce-a02b-ff2e33a01217-kube-api-access-vd467\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.925677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.925639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428df5de-7c4b-47ce-a02b-ff2e33a01217-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.926238 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.926214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428df5de-7c4b-47ce-a02b-ff2e33a01217-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.928189 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.928165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428df5de-7c4b-47ce-a02b-ff2e33a01217-metrics-certs\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.928285 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.928262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428df5de-7c4b-47ce-a02b-ff2e33a01217-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:06.935986 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:06.935959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd467\" (UniqueName: \"kubernetes.io/projected/428df5de-7c4b-47ce-a02b-ff2e33a01217-kube-api-access-vd467\") pod \"maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz\" (UID: \"428df5de-7c4b-47ce-a02b-ff2e33a01217\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:07.013558 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.013521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:07.309565 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.309539    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"]
Sep 16 13:21:07.311273 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:21:07.311245    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428df5de_7c4b_47ce_a02b_ff2e33a01217.slice/crio-f4abeb0136b87464eac0b2e7f6f44ffdce672f8433411d0a7bd0cefeb0bc4aeb WatchSource:0}: Error finding container f4abeb0136b87464eac0b2e7f6f44ffdce672f8433411d0a7bd0cefeb0bc4aeb: Status 404 returned error can't find the container with id f4abeb0136b87464eac0b2e7f6f44ffdce672f8433411d0a7bd0cefeb0bc4aeb
Sep 16 13:21:07.430629 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.430588    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"]
Sep 16 13:21:07.433768 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.433746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.436087 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.436067    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-15da7b-openshift-default-dockercfg-wpvrv\""
Sep 16 13:21:07.447112 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.447082    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"]
Sep 16 13:21:07.533057 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-socket\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.533234 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-istio-token\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.533234 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c9342-1c8e-43d2-8a7b-accae753f322-istio-podinfo\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.533234 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-credential-socket\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.533417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-certs\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.533417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-envoy\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.533417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c9342-1c8e-43d2-8a7b-accae753f322-istiod-ca-cert\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.533565 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-data\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.533565 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.533470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsqv\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-kube-api-access-vrsqv\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634392 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-istio-token\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634392 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c9342-1c8e-43d2-8a7b-accae753f322-istio-podinfo\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634392 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-credential-socket\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-certs\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-envoy\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c9342-1c8e-43d2-8a7b-accae753f322-istiod-ca-cert\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-data\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsqv\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-kube-api-access-vrsqv\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.634694 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-socket\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.635090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.634968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-credential-socket\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.635090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.635067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-socket\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.635252 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.635218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-data\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.635337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.635322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-certs\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.635435 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.635411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c9342-1c8e-43d2-8a7b-accae753f322-istiod-ca-cert\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.636917 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.636895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-envoy\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.637043 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.637003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c9342-1c8e-43d2-8a7b-accae753f322-istio-podinfo\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.652714 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.652690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-istio-token\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.652940 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.652882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsqv\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-kube-api-access-vrsqv\") pod \"e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") " pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.746572 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.746539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:07.790896 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.790752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz" event={"ID":"428df5de-7c4b-47ce-a02b-ff2e33a01217","Type":"ContainerStarted","Data":"c34389f408668ab3c5f329b684c4bf0003e7ac7be487f44f2306c374756d94d6"}
Sep 16 13:21:07.790896 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.790817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz" event={"ID":"428df5de-7c4b-47ce-a02b-ff2e33a01217","Type":"ContainerStarted","Data":"f4abeb0136b87464eac0b2e7f6f44ffdce672f8433411d0a7bd0cefeb0bc4aeb"}
Sep 16 13:21:07.791153 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.791119    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:07.794159 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.794116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq" event={"ID":"1a80e599-65c3-4d9a-bdb2-871965459827","Type":"ContainerStarted","Data":"189917999b7ab1f571512dfbce14108ca179566702c51b51e0c320fbd2ff11a5"}
Sep 16 13:21:07.794271 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.794181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq" event={"ID":"1a80e599-65c3-4d9a-bdb2-871965459827","Type":"ContainerStarted","Data":"dab2de9ae76eb090a79e657c4e4308ef5cc6ad4f802b54cf952d437f7ea8fa08"}
Sep 16 13:21:07.794271 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.794227    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:07.809566 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.809102    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz" podStartSLOduration=1.452727361 podStartE2EDuration="1.809077274s" totalImagesPullingTime="356.349913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:07.808418657 +0000 UTC m=+739.231160261" watchObservedRunningTime="2026-09-16 13:21:07.809077274 +0000 UTC m=+739.231818854"
Sep 16 13:21:07.827366 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:07.827306    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq" podStartSLOduration=2.407343589 podStartE2EDuration="2.827288175s" totalImagesPullingTime="419.944586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:07.825878315 +0000 UTC m=+739.248619895" watchObservedRunningTime="2026-09-16 13:21:07.827288175 +0000 UTC m=+739.250029754"
Sep 16 13:21:08.046999 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:08.046971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"]
Sep 16 13:21:08.048927 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:21:08.048897    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960c9342_1c8e_43d2_8a7b_accae753f322.slice/crio-5ab725b6f2a2c13bc6655e83d667fe55e449b69e97c42cd3690f0847921b3826 WatchSource:0}: Error finding container 5ab725b6f2a2c13bc6655e83d667fe55e449b69e97c42cd3690f0847921b3826: Status 404 returned error can't find the container with id 5ab725b6f2a2c13bc6655e83d667fe55e449b69e97c42cd3690f0847921b3826
Sep 16 13:21:08.051262 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:08.051229    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:08.051354 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:08.051293    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:08.051354 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:08.051323    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:08.805204 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:08.805167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" event={"ID":"960c9342-1c8e-43d2-8a7b-accae753f322","Type":"ContainerStarted","Data":"3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9"}
Sep 16 13:21:08.805621 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:08.805229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" event={"ID":"960c9342-1c8e-43d2-8a7b-accae753f322","Type":"ContainerStarted","Data":"5ab725b6f2a2c13bc6655e83d667fe55e449b69e97c42cd3690f0847921b3826"}
Sep 16 13:21:08.829333 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:08.829284    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" podStartSLOduration=1.829266386 podStartE2EDuration="1.829266386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:08.826953475 +0000 UTC m=+740.249695056" watchObservedRunningTime="2026-09-16 13:21:08.829266386 +0000 UTC m=+740.252007964"
Sep 16 13:21:09.747692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:09.747658    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:09.753503 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:09.753473    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:09.810909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:09.810871    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:09.811760 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:09.811741    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:12.799790 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:12.799760    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7"
Sep 16 13:21:13.810695 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:13.810664    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz"
Sep 16 13:21:13.811133 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:13.810822    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq"
Sep 16 13:21:17.228435 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.228390    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"]
Sep 16 13:21:17.240774 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.240740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.243877 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.243635    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-15da7b-serving-cert\""
Sep 16 13:21:17.244637 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.244600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"]
Sep 16 13:21:17.251663 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.251628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.251784 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.251676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217537df-4321-4e99-928a-c8f6ffe3320c-openshift-service-ca\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.251784 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.251738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-metrics-certs\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.251925 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.251793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42z9j\" (UniqueName: \"kubernetes.io/projected/217537df-4321-4e99-928a-c8f6ffe3320c-kube-api-access-42z9j\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.355721 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.353442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.355721 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.353503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217537df-4321-4e99-928a-c8f6ffe3320c-openshift-service-ca\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.355721 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.353563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-metrics-certs\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.355721 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.353615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42z9j\" (UniqueName: \"kubernetes.io/projected/217537df-4321-4e99-928a-c8f6ffe3320c-kube-api-access-42z9j\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.355721 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:21:17.354146    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-serving-cert: secret "maas-api-e2e-autores-15da7b-serving-cert" not found
Sep 16 13:21:17.355721 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:21:17.354219    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls podName:217537df-4321-4e99-928a-c8f6ffe3320c nodeName:}" failed. No retries permitted until 2026-09-16 13:21:17.854199731 +0000 UTC m=+749.276941293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls") pod "maas-api-e2e-autores-15da7b-7986544d84-vr8b5" (UID: "217537df-4321-4e99-928a-c8f6ffe3320c") : secret "maas-api-e2e-autores-15da7b-serving-cert" not found
Sep 16 13:21:17.355721 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.355341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217537df-4321-4e99-928a-c8f6ffe3320c-openshift-service-ca\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.358701 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.358656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-metrics-certs\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.363656 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.363631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42z9j\" (UniqueName: \"kubernetes.io/projected/217537df-4321-4e99-928a-c8f6ffe3320c-kube-api-access-42z9j\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.860319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.860215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:17.862712 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:17.862686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls\") pod \"maas-api-e2e-autores-15da7b-7986544d84-vr8b5\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:18.157333 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.157227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:18.305272 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.305042    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"]
Sep 16 13:21:18.309988 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.309956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.312312 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.312274    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c71ad7-openshift-default-dockercfg-thb2n\""
Sep 16 13:21:18.324253 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.324222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"]
Sep 16 13:21:18.367895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.367838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-credential-socket\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.368071 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.367945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvlgw\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-kube-api-access-nvlgw\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.368071 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.368054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-certs\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.368157 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.368097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-envoy\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.368202 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.368157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-data\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.368202 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.368191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-podinfo\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.368280 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.368232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-socket\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.368325 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.368312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-token\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.368374 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.368360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istiod-ca-cert\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.453424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.452276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"]
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-credential-socket\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvlgw\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-kube-api-access-nvlgw\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-certs\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-envoy\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-data\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-podinfo\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-socket\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-token\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.470532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istiod-ca-cert\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471607 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.471376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istiod-ca-cert\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.471682 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.471653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-data\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.473790 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.471828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-credential-socket\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.473790 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.472117    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-certs\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.473790 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.472431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-socket\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.473790 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:21:18.472981    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217537df_4321_4e99_928a_c8f6ffe3320c.slice/crio-5428f52f3daa95f9370f2eaeeb40322704721072e0472eaad52a5beff0b7b0bc WatchSource:0}: Error finding container 5428f52f3daa95f9370f2eaeeb40322704721072e0472eaad52a5beff0b7b0bc: Status 404 returned error can't find the container with id 5428f52f3daa95f9370f2eaeeb40322704721072e0472eaad52a5beff0b7b0bc
Sep 16 13:21:18.476090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.475961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-podinfo\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.477033 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.476990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-envoy\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.487390 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.486748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-token\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.491196 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.491131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvlgw\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-kube-api-access-nvlgw\") pod \"e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") " pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.630434 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.630384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:18.792134 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.792027    2577 patch_prober.go:28] interesting pod/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 13:21:18.792134 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.792095    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 13:21:18.879104 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.879060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5" event={"ID":"217537df-4321-4e99-928a-c8f6ffe3320c","Type":"ContainerStarted","Data":"5428f52f3daa95f9370f2eaeeb40322704721072e0472eaad52a5beff0b7b0bc"}
Sep 16 13:21:18.925781 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.925733    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"]
Sep 16 13:21:18.937780 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.937733    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:18.937950 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.937820    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:18.937950 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:18.937880    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:19.889127 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:19.889086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5" event={"ID":"217537df-4321-4e99-928a-c8f6ffe3320c","Type":"ContainerStarted","Data":"543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde"}
Sep 16 13:21:19.889600 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:19.889273    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:19.891340 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:19.891303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" event={"ID":"5ac52012-5e9d-4c07-830f-b36b444cb8eb","Type":"ContainerStarted","Data":"d72d880776a7993c81e6ad72b8007365c56f177165c91c13500f35926e3a575b"}
Sep 16 13:21:19.891478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:19.891366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" event={"ID":"5ac52012-5e9d-4c07-830f-b36b444cb8eb","Type":"ContainerStarted","Data":"1a39b8fbd86247648a0f38d7bd806e7c0987585852a4bdc8dc892fd737bc4c92"}
Sep 16 13:21:19.916760 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:19.916701    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5" podStartSLOduration=2.451092655 podStartE2EDuration="2.916685258s" totalImagesPullingTime="465.592603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:19.913833111 +0000 UTC m=+751.336574705" watchObservedRunningTime="2026-09-16 13:21:19.916685258 +0000 UTC m=+751.339426836"
Sep 16 13:21:19.947531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:19.947472    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" podStartSLOduration=1.947455502 podStartE2EDuration="1.947455502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:19.940097391 +0000 UTC m=+751.362838970" watchObservedRunningTime="2026-09-16 13:21:19.947455502 +0000 UTC m=+751.370197081"
Sep 16 13:21:20.631605 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:20.631564    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:20.637409 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:20.637376    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:20.898971 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:20.898881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:20.901028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:20.899655    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:21.905798 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.905763    2577 generic.go:356] "Generic (PLEG): container finished" podID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerID="f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183" exitCode=0
Sep 16 13:21:21.906355 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.905842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" event={"ID":"ab5b0ea4-eb39-493b-8c35-1a9e3918726b","Type":"ContainerDied","Data":"f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183"}
Sep 16 13:21:21.906455 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.906437    2577 scope.go:117] "RemoveContainer" containerID="f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183"
Sep 16 13:21:21.907521 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.907502    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerID="1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba" exitCode=0
Sep 16 13:21:21.907597 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.907574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" event={"ID":"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9","Type":"ContainerDied","Data":"1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba"}
Sep 16 13:21:21.907824 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.907796    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:21.907981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.907885    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:21.907981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.907919    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:21.908097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.908069    2577 scope.go:117] "RemoveContainer" containerID="1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba"
Sep 16 13:21:21.909040 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.909013    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:21.909150 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.909072    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:21.909150 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:21.909120    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:22.916073 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:22.916032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" event={"ID":"ab5b0ea4-eb39-493b-8c35-1a9e3918726b","Type":"ContainerStarted","Data":"c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2"}
Sep 16 13:21:22.918242 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:22.918216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" event={"ID":"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9","Type":"ContainerStarted","Data":"a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7"}
Sep 16 13:21:23.664291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:23.664244    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:21:23.664506 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:23.664415    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 16 13:21:23.664506 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:23.664477    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 16 13:21:23.874966 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:23.874911    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:21:23.874966 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:23.874963    2577 patch_prober.go:28] interesting pod/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 13:21:23.875245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:23.875017    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 13:21:24.664487 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:24.664443    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 16 13:21:24.665128 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:24.665092    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 16 13:21:24.875058 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:24.875013    2577 patch_prober.go:28] interesting pod/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 13:21:24.875280 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:24.875101    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 13:21:25.664305 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:25.664266    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 16 13:21:25.665028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:25.664596    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 16 13:21:25.875120 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:25.875069    2577 patch_prober.go:28] interesting pod/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 13:21:25.875339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:25.875148    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 13:21:25.908907 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:25.908825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:21:26.664484 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:26.664443    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 16 13:21:26.665249 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:26.664809    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 16 13:21:26.874691 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:26.874651    2577 patch_prober.go:28] interesting pod/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 13:21:26.875001 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:26.874963    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 13:21:27.663668 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:27.663618    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 16 13:21:27.663905 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:27.663694    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 16 13:21:27.874743 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:27.874706    2577 patch_prober.go:28] interesting pod/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 13:21:27.875768 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:27.875363    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 13:21:28.664025 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:28.663923    2577 patch_prober.go:28] interesting pod/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 16 13:21:28.664025 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:28.663993    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 16 13:21:28.946529 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:28.946491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:21:28.948912 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:28.948888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:21:28.957008 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:28.956983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:21:29.668388 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:29.668354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:21:29.668672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:29.668649    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:21:29.669449 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:29.669430    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:21:35.045199 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:35.045162    2577 generic.go:356] "Generic (PLEG): container finished" podID="960c9342-1c8e-43d2-8a7b-accae753f322" containerID="3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9" exitCode=0
Sep 16 13:21:35.045728 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:35.045239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" event={"ID":"960c9342-1c8e-43d2-8a7b-accae753f322","Type":"ContainerDied","Data":"3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9"}
Sep 16 13:21:35.045900 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:35.045883    2577 scope.go:117] "RemoveContainer" containerID="3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9"
Sep 16 13:21:35.047168 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:35.047142    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:35.047244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:35.047215    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:35.047282 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:35.047246    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:36.057534 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:36.057496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" event={"ID":"960c9342-1c8e-43d2-8a7b-accae753f322","Type":"ContainerStarted","Data":"f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27"}
Sep 16 13:21:36.747023 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:36.746983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:36.748443 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:36.748412    2577 patch_prober.go:28] interesting pod/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:21:36.748663 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:36.748635    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 13:21:37.747480 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:37.747436    2577 patch_prober.go:28] interesting pod/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:21:37.748102 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:37.748070    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 13:21:38.747460 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:38.747003    2577 patch_prober.go:28] interesting pod/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:21:38.747460 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:38.747067    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 13:21:39.747570 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:39.747103    2577 patch_prober.go:28] interesting pod/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:21:39.747570 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:39.747176    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 13:21:39.811715 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:39.811670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:40.747349 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:40.747307    2577 patch_prober.go:28] interesting pod/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:21:40.747664 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:40.747633    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 13:21:41.765489 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:41.765455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:41.767357 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:41.767333    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:21:47.154033 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:47.153934    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerID="d72d880776a7993c81e6ad72b8007365c56f177165c91c13500f35926e3a575b" exitCode=0
Sep 16 13:21:47.154033 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:47.153961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" event={"ID":"5ac52012-5e9d-4c07-830f-b36b444cb8eb","Type":"ContainerDied","Data":"d72d880776a7993c81e6ad72b8007365c56f177165c91c13500f35926e3a575b"}
Sep 16 13:21:47.154630 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:47.154460    2577 scope.go:117] "RemoveContainer" containerID="d72d880776a7993c81e6ad72b8007365c56f177165c91c13500f35926e3a575b"
Sep 16 13:21:47.156049 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:47.156016    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:47.156146 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:47.156108    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:47.156192 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:47.156159    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:48.164506 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:48.164471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" event={"ID":"5ac52012-5e9d-4c07-830f-b36b444cb8eb","Type":"ContainerStarted","Data":"7958ba326e44a017f13e2d744bb32fdcfe9bbddd8dd24406b9440b6b919e7047"}
Sep 16 13:21:48.631524 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:48.631484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:48.632697 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:48.632665    2577 patch_prober.go:28] interesting pod/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 13:21:48.632957 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:48.632927    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 13:21:49.631783 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:49.631735    2577 patch_prober.go:28] interesting pod/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 13:21:49.632299 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:49.631819    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 13:21:50.631763 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:50.631717    2577 patch_prober.go:28] interesting pod/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 13:21:50.631989 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:50.631819    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 13:21:50.899006 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:50.898920    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:51.631525 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:51.631080    2577 patch_prober.go:28] interesting pod/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 13:21:51.631525 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:51.631143    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 13:21:52.631476 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:52.631420    2577 patch_prober.go:28] interesting pod/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 13:21:52.632003 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:52.631514    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 13:21:53.637028 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:53.636996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:53.638390 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:53.638363    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:21:55.039894 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.039830    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"]
Sep 16 13:21:55.044240 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.044217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.046636 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.046611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-508c5fc8-gw-openshift-default-dockercfg-j5544\""
Sep 16 13:21:55.056996 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.056969    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"]
Sep 16 13:21:55.177109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-certs\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.177109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9kj\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-kube-api-access-px9kj\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.177327 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-data\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.177327 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-socket\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.177327 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-credential-socket\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.177447 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-token\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.177447 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-envoy\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.177447 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4d6672-324f-40bd-80a4-1e441d19e1da-istiod-ca-cert\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.177554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.177456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-podinfo\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.278591 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-certs\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.278776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9kj\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-kube-api-access-px9kj\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.278776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-data\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.278776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-socket\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.278776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-credential-socket\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.278776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-token\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.278776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-envoy\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.279124 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4d6672-324f-40bd-80a4-1e441d19e1da-istiod-ca-cert\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.279124 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.278835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-podinfo\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.279226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.279114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-data\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.279226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.279180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-credential-socket\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.279347 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.279326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-socket\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.279417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.279401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-certs\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.279737 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.279717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4d6672-324f-40bd-80a4-1e441d19e1da-istiod-ca-cert\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.281298 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.281275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-envoy\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.281617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.281600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-podinfo\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.286239 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.286216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9kj\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-kube-api-access-px9kj\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.286350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.286243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-token\") pod \"e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") " pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.356611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.356521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:55.635059 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.635030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"]
Sep 16 13:21:55.635821 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:21:55.635794    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4d6672_324f_40bd_80a4_1e441d19e1da.slice/crio-936dc1eda8acfb7f9ea401458dd6a0a3013283c7d97ecae24667a811d36b8a9c WatchSource:0}: Error finding container 936dc1eda8acfb7f9ea401458dd6a0a3013283c7d97ecae24667a811d36b8a9c: Status 404 returned error can't find the container with id 936dc1eda8acfb7f9ea401458dd6a0a3013283c7d97ecae24667a811d36b8a9c
Sep 16 13:21:55.638606 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.638572    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:55.638705 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.638652    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:55.638769 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:55.638704    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:21:56.212232 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:56.212195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" event={"ID":"fb4d6672-324f-40bd-80a4-1e441d19e1da","Type":"ContainerStarted","Data":"2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e"}
Sep 16 13:21:56.212623 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:56.212252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" event={"ID":"fb4d6672-324f-40bd-80a4-1e441d19e1da","Type":"ContainerStarted","Data":"936dc1eda8acfb7f9ea401458dd6a0a3013283c7d97ecae24667a811d36b8a9c"}
Sep 16 13:21:56.235806 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:56.235737    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" podStartSLOduration=1.235716606 podStartE2EDuration="1.235716606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:56.231573899 +0000 UTC m=+787.654315491" watchObservedRunningTime="2026-09-16 13:21:56.235716606 +0000 UTC m=+787.658458185"
Sep 16 13:21:56.357341 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:56.357302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:56.362802 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:56.362777    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:57.217543 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:57.217505    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:21:57.218541 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:21:57.218524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:22:22.361908 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:22.361871    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerID="2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e" exitCode=0
Sep 16 13:22:22.362345 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:22.361942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" event={"ID":"fb4d6672-324f-40bd-80a4-1e441d19e1da","Type":"ContainerDied","Data":"2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e"}
Sep 16 13:22:22.362419 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:22.362405    2577 scope.go:117] "RemoveContainer" containerID="2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e"
Sep 16 13:22:22.363593 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:22.363567    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:22:22.363681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:22.363631    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:22:22.363681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:22.363667    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:22:23.369914 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:23.369881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" event={"ID":"fb4d6672-324f-40bd-80a4-1e441d19e1da","Type":"ContainerStarted","Data":"c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80"}
Sep 16 13:22:24.357417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:24.357378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:22:24.358309 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:24.358275    2577 patch_prober.go:28] interesting pod/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 13:22:24.358458 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:24.358332    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 13:22:25.357019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:25.356976    2577 patch_prober.go:28] interesting pod/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 13:22:25.357598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:25.357562    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 13:22:26.357652 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:26.357618    2577 patch_prober.go:28] interesting pod/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 13:22:26.358246 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:26.358206    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 13:22:27.217996 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:27.217957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:22:27.357035 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:27.356992    2577 patch_prober.go:28] interesting pod/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 13:22:27.357340 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:27.357306    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 13:22:28.361335 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:28.361298    2577 patch_prober.go:28] interesting pod/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 13:22:28.361966 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:28.361932    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 13:22:29.293099 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:29.293053    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"]
Sep 16 13:22:29.293315 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:29.293278    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy" containerID="cri-o://c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80" gracePeriod=30
Sep 16 13:22:34.660295 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.660268    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:22:34.820990 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.820905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-token\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.820990 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.820951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-envoy\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.821210 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-credential-socket\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.821210 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821060    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-socket\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.821210 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821082    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-certs\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.821210 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821162    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-data\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.821210 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px9kj\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-kube-api-access-px9kj\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.821468 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821225    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4d6672-324f-40bd-80a4-1e441d19e1da-istiod-ca-cert\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.821468 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-podinfo\") pod \"fb4d6672-324f-40bd-80a4-1e441d19e1da\" (UID: \"fb4d6672-324f-40bd-80a4-1e441d19e1da\") "
Sep 16 13:22:34.821468 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:34.821468 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:34.821468 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:34.821662 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:34.821662 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4d6672-324f-40bd-80a4-1e441d19e1da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:22:34.821791 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821776    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:34.821876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821795    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:34.821876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821805    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:34.821876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821814    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:34.821876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.821822    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4d6672-324f-40bd-80a4-1e441d19e1da-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:34.823506 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.823478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-kube-api-access-px9kj" (OuterVolumeSpecName: "kube-api-access-px9kj") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "kube-api-access-px9kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:22:34.823611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.823573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:22:34.823680 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.823658    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:34.823717 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.823672    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb4d6672-324f-40bd-80a4-1e441d19e1da" (UID: "fb4d6672-324f-40bd-80a4-1e441d19e1da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:22:34.923126 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.923083    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9kj\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-kube-api-access-px9kj\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:34.923126 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.923117    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:34.923126 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.923130    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:34.923357 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:34.923141    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4d6672-324f-40bd-80a4-1e441d19e1da-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:22:35.449672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.449643    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerID="c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80" exitCode=0
Sep 16 13:22:35.449884 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.449702    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"
Sep 16 13:22:35.449884 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.449730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" event={"ID":"fb4d6672-324f-40bd-80a4-1e441d19e1da","Type":"ContainerDied","Data":"c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80"}
Sep 16 13:22:35.449884 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.449787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7" event={"ID":"fb4d6672-324f-40bd-80a4-1e441d19e1da","Type":"ContainerDied","Data":"936dc1eda8acfb7f9ea401458dd6a0a3013283c7d97ecae24667a811d36b8a9c"}
Sep 16 13:22:35.449884 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.449819    2577 scope.go:117] "RemoveContainer" containerID="c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80"
Sep 16 13:22:35.460010 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.459989    2577 scope.go:117] "RemoveContainer" containerID="2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e"
Sep 16 13:22:35.468706 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.468685    2577 scope.go:117] "RemoveContainer" containerID="c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80"
Sep 16 13:22:35.468998 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:22:35.468978    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80\": container with ID starting with c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80 not found: ID does not exist" containerID="c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80"
Sep 16 13:22:35.469054 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.469008    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80"} err="failed to get container status \"c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80\": rpc error: code = NotFound desc = could not find container \"c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80\": container with ID starting with c0b80efc2bb1782c634c998c52b2b9ef2d21d360a4f4e2a65230f75f0bd3dc80 not found: ID does not exist"
Sep 16 13:22:35.469054 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.469027    2577 scope.go:117] "RemoveContainer" containerID="2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e"
Sep 16 13:22:35.469255 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:22:35.469241    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e\": container with ID starting with 2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e not found: ID does not exist" containerID="2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e"
Sep 16 13:22:35.469313 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.469259    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e"} err="failed to get container status \"2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e\": rpc error: code = NotFound desc = could not find container \"2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e\": container with ID starting with 2d5c3892cd79b5dcba6d2d7b189b5307d93d9d04d695cdedf9742b39611b6e3e not found: ID does not exist"
Sep 16 13:22:35.476722 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.476699    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"]
Sep 16 13:22:35.484393 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:35.484367    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-508c5fc8-gw-openshift-default-d9f66878d-hrlj7"]
Sep 16 13:22:37.118101 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:22:37.118066    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" path="/var/lib/kubelet/pods/fb4d6672-324f-40bd-80a4-1e441d19e1da/volumes"
Sep 16 13:23:17.602458 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.602413    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"]
Sep 16 13:23:17.603027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.602916    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy"
Sep 16 13:23:17.603027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.602929    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy"
Sep 16 13:23:17.603027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.602958    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy"
Sep 16 13:23:17.603027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.602964    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy"
Sep 16 13:23:17.603226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.603039    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy"
Sep 16 13:23:17.603226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.603051    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4d6672-324f-40bd-80a4-1e441d19e1da" containerName="istio-proxy"
Sep 16 13:23:17.605456 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.605428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.608355 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.608332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9615f1d6-openshift-default-dockercfg-mbj9z\""
Sep 16 13:23:17.619317 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.619290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"]
Sep 16 13:23:17.675558 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-certs\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.675558 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c04164-f683-4a4b-ad32-b0b7714172d5-istiod-ca-cert\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.675767 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-podinfo\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.675767 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-token\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.675767 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-credential-socket\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.675902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-data\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.675902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wk4k\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-kube-api-access-9wk4k\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.675902 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-envoy\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.676000 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.675927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-socket\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777283 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wk4k\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-kube-api-access-9wk4k\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777283 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-envoy\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777541 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-socket\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777541 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-certs\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777541 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c04164-f683-4a4b-ad32-b0b7714172d5-istiod-ca-cert\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777541 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-podinfo\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777541 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-token\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777889 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-credential-socket\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.777889 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-data\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.778034 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-certs\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.778034 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.777984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-socket\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.778220 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.778164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-credential-socket\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.778220 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.778176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-data\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.778388 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.778313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c04164-f683-4a4b-ad32-b0b7714172d5-istiod-ca-cert\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.779912 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.779888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-envoy\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.780008 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.779977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-podinfo\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.784935 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.784907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-token\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.784935 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.784922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wk4k\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-kube-api-access-9wk4k\") pod \"e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") " pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:17.920823 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:17.920736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:18.213292 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:18.213244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"]
Sep 16 13:23:18.215460 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:23:18.215431    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c04164_f683_4a4b_ad32_b0b7714172d5.slice/crio-6f1dba1929ba0760de7e9f1d8cee2e8204491068568559ceb731364e3730e08d WatchSource:0}: Error finding container 6f1dba1929ba0760de7e9f1d8cee2e8204491068568559ceb731364e3730e08d: Status 404 returned error can't find the container with id 6f1dba1929ba0760de7e9f1d8cee2e8204491068568559ceb731364e3730e08d
Sep 16 13:23:18.218228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:18.218192    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:23:18.218319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:18.218259    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:23:18.218319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:18.218288    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:23:18.700371 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:18.700337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" event={"ID":"37c04164-f683-4a4b-ad32-b0b7714172d5","Type":"ContainerStarted","Data":"50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b"}
Sep 16 13:23:18.700759 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:18.700392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" event={"ID":"37c04164-f683-4a4b-ad32-b0b7714172d5","Type":"ContainerStarted","Data":"6f1dba1929ba0760de7e9f1d8cee2e8204491068568559ceb731364e3730e08d"}
Sep 16 13:23:18.724226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:18.724173    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podStartSLOduration=1.7241572010000001 podStartE2EDuration="1.724157201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:18.721240242 +0000 UTC m=+870.143981822" watchObservedRunningTime="2026-09-16 13:23:18.724157201 +0000 UTC m=+870.146898856"
Sep 16 13:23:18.921223 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:18.921186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:19.926516 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:19.926487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:20.714245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:20.714209    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:20.715223 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:20.715200    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:43.853623 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:43.853589    2577 generic.go:356] "Generic (PLEG): container finished" podID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerID="50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b" exitCode=0
Sep 16 13:23:43.854149 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:43.853666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" event={"ID":"37c04164-f683-4a4b-ad32-b0b7714172d5","Type":"ContainerDied","Data":"50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b"}
Sep 16 13:23:43.854260 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:43.854243    2577 scope.go:117] "RemoveContainer" containerID="50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b"
Sep 16 13:23:43.855460 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:43.855431    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:23:43.855521 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:43.855501    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:23:43.855564 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:43.855538    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:23:44.862069 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:44.862031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" event={"ID":"37c04164-f683-4a4b-ad32-b0b7714172d5","Type":"ContainerStarted","Data":"44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe"}
Sep 16 13:23:44.921670 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:44.921635    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:44.921907 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:44.921884    2577 patch_prober.go:28] interesting pod/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 16 13:23:44.921969 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:44.921946    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 16 13:23:45.921374 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:45.921336    2577 patch_prober.go:28] interesting pod/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 16 13:23:45.921834 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:45.921407    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 16 13:23:46.921454 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:46.921420    2577 patch_prober.go:28] interesting pod/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 16 13:23:46.921837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:46.921487    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 16 13:23:47.921925 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:47.921889    2577 patch_prober.go:28] interesting pod/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 16 13:23:47.922308 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:47.921953    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 16 13:23:48.921418 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:48.921379    2577 patch_prober.go:28] interesting pod/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 16 13:23:48.921572 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:48.921433    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 16 13:23:49.921970 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:49.921920    2577 patch_prober.go:28] interesting pod/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 16 13:23:49.922467 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:49.921997    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 16 13:23:50.714390 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:50.714352    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:50.925514 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:50.925477    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:50.926527 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:50.926502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:23:56.497367 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.497331    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"]
Sep 16 13:23:56.497789 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.497528    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5" podUID="217537df-4321-4e99-928a-c8f6ffe3320c" containerName="maas-api" containerID="cri-o://543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde" gracePeriod=30
Sep 16 13:23:56.768358 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.768333    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:23:56.916031 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.915980    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42z9j\" (UniqueName: \"kubernetes.io/projected/217537df-4321-4e99-928a-c8f6ffe3320c-kube-api-access-42z9j\") pod \"217537df-4321-4e99-928a-c8f6ffe3320c\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") "
Sep 16 13:23:56.916419 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.916360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217537df-4321-4e99-928a-c8f6ffe3320c-openshift-service-ca\") pod \"217537df-4321-4e99-928a-c8f6ffe3320c\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") "
Sep 16 13:23:56.916832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.916762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-metrics-certs\") pod \"217537df-4321-4e99-928a-c8f6ffe3320c\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") "
Sep 16 13:23:56.917125 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.917071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls\") pod \"217537df-4321-4e99-928a-c8f6ffe3320c\" (UID: \"217537df-4321-4e99-928a-c8f6ffe3320c\") "
Sep 16 13:23:56.917560 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.917457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217537df-4321-4e99-928a-c8f6ffe3320c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "217537df-4321-4e99-928a-c8f6ffe3320c" (UID: "217537df-4321-4e99-928a-c8f6ffe3320c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:23:56.917888 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.917781    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217537df-4321-4e99-928a-c8f6ffe3320c-openshift-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:23:56.923218 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.922860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217537df-4321-4e99-928a-c8f6ffe3320c-kube-api-access-42z9j" (OuterVolumeSpecName: "kube-api-access-42z9j") pod "217537df-4321-4e99-928a-c8f6ffe3320c" (UID: "217537df-4321-4e99-928a-c8f6ffe3320c"). InnerVolumeSpecName "kube-api-access-42z9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:23:56.924008 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.923972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "217537df-4321-4e99-928a-c8f6ffe3320c" (UID: "217537df-4321-4e99-928a-c8f6ffe3320c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:23:56.934352 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.933284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "217537df-4321-4e99-928a-c8f6ffe3320c" (UID: "217537df-4321-4e99-928a-c8f6ffe3320c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:23:56.949043 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.949005    2577 generic.go:356] "Generic (PLEG): container finished" podID="217537df-4321-4e99-928a-c8f6ffe3320c" containerID="543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde" exitCode=0
Sep 16 13:23:56.949197 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.949061    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"
Sep 16 13:23:56.949197 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.949099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5" event={"ID":"217537df-4321-4e99-928a-c8f6ffe3320c","Type":"ContainerDied","Data":"543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde"}
Sep 16 13:23:56.949197 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.949138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5" event={"ID":"217537df-4321-4e99-928a-c8f6ffe3320c","Type":"ContainerDied","Data":"5428f52f3daa95f9370f2eaeeb40322704721072e0472eaad52a5beff0b7b0bc"}
Sep 16 13:23:56.949197 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.949165    2577 scope.go:117] "RemoveContainer" containerID="543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde"
Sep 16 13:23:56.960193 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.960169    2577 scope.go:117] "RemoveContainer" containerID="543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde"
Sep 16 13:23:56.960502 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:23:56.960476    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde\": container with ID starting with 543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde not found: ID does not exist" containerID="543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde"
Sep 16 13:23:56.960594 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.960517    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde"} err="failed to get container status \"543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde\": rpc error: code = NotFound desc = could not find container \"543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde\": container with ID starting with 543d9f5233e91e46efceb9934c25bfa8cd546a42f13559eac300dcf95c89bbde not found: ID does not exist"
Sep 16 13:23:56.972582 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.972549    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"]
Sep 16 13:23:56.977592 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:56.977567    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-15da7b-7986544d84-vr8b5"]
Sep 16 13:23:57.018405 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:57.018342    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-maas-api-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:23:57.018405 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:57.018366    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42z9j\" (UniqueName: \"kubernetes.io/projected/217537df-4321-4e99-928a-c8f6ffe3320c-kube-api-access-42z9j\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:23:57.018405 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:57.018375    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217537df-4321-4e99-928a-c8f6ffe3320c-metrics-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:23:57.110697 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:23:57.110654    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217537df-4321-4e99-928a-c8f6ffe3320c" path="/var/lib/kubelet/pods/217537df-4321-4e99-928a-c8f6ffe3320c/volumes"
Sep 16 13:24:03.349070 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:03.349032    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"]
Sep 16 13:24:03.349567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:03.349292    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy" containerID="cri-o://44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe" gracePeriod=30
Sep 16 13:24:05.928802 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:05.928762    2577 patch_prober.go:28] interesting pod/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 16 13:24:05.929201 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:05.928828    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:08.610686 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.610626    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:24:08.742845 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.742815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-envoy\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.742891    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-podinfo\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.742932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-token\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-credential-socket\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-certs\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743069    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-socket\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743356 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743098    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wk4k\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-kube-api-access-9wk4k\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743356 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-data\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743376    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:08.743462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743389    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:08.743462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c04164-f683-4a4b-ad32-b0b7714172d5-istiod-ca-cert\") pod \"37c04164-f683-4a4b-ad32-b0b7714172d5\" (UID: \"37c04164-f683-4a4b-ad32-b0b7714172d5\") "
Sep 16 13:24:08.743626 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:08.743798 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:08.743918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743842    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:08.743918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743887    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:08.743918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743898    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:08.743918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743906    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:08.744159 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.743959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c04164-f683-4a4b-ad32-b0b7714172d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:24:08.745211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.745190    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:08.745318 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.745236    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:24:08.745608 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.745589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-kube-api-access-9wk4k" (OuterVolumeSpecName: "kube-api-access-9wk4k") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "kube-api-access-9wk4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:08.745672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.745613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "37c04164-f683-4a4b-ad32-b0b7714172d5" (UID: "37c04164-f683-4a4b-ad32-b0b7714172d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:08.844775 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.844740    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:08.844775 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.844768    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wk4k\" (UniqueName: \"kubernetes.io/projected/37c04164-f683-4a4b-ad32-b0b7714172d5-kube-api-access-9wk4k\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:08.844775 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.844780    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c04164-f683-4a4b-ad32-b0b7714172d5-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:08.845046 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.844790    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:08.845046 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:08.844799    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c04164-f683-4a4b-ad32-b0b7714172d5-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:09.016839 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.016801    2577 generic.go:356] "Generic (PLEG): container finished" podID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerID="44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe" exitCode=0
Sep 16 13:24:09.017047 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.016845    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" event={"ID":"37c04164-f683-4a4b-ad32-b0b7714172d5","Type":"ContainerDied","Data":"44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe"}
Sep 16 13:24:09.017047 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.016889    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"
Sep 16 13:24:09.017047 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.016939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn" event={"ID":"37c04164-f683-4a4b-ad32-b0b7714172d5","Type":"ContainerDied","Data":"6f1dba1929ba0760de7e9f1d8cee2e8204491068568559ceb731364e3730e08d"}
Sep 16 13:24:09.017047 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.016970    2577 scope.go:117] "RemoveContainer" containerID="44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe"
Sep 16 13:24:09.027298 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.027278    2577 scope.go:117] "RemoveContainer" containerID="50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b"
Sep 16 13:24:09.040363 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.040343    2577 scope.go:117] "RemoveContainer" containerID="44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe"
Sep 16 13:24:09.040731 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:24:09.040712    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe\": container with ID starting with 44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe not found: ID does not exist" containerID="44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe"
Sep 16 13:24:09.040816 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.040739    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe"} err="failed to get container status \"44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe\": rpc error: code = NotFound desc = could not find container \"44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe\": container with ID starting with 44faa6252c3eba01f3ac045dc06f45a970c2f3c2ed8fbbb24b1a4f7de8efd3fe not found: ID does not exist"
Sep 16 13:24:09.040816 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.040769    2577 scope.go:117] "RemoveContainer" containerID="50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b"
Sep 16 13:24:09.041126 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:24:09.041085    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b\": container with ID starting with 50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b not found: ID does not exist" containerID="50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b"
Sep 16 13:24:09.041269 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.041129    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b"} err="failed to get container status \"50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b\": rpc error: code = NotFound desc = could not find container \"50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b\": container with ID starting with 50903138ec6e4b478238c46564ad2b807adc4f94dabd7a4720cece375f93ba6b not found: ID does not exist"
Sep 16 13:24:09.042491 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.042469    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"]
Sep 16 13:24:09.044662 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.044644    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9615f1d6-openshift-default-7b658d9c6-jvszn"]
Sep 16 13:24:09.112474 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:09.112442    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" path="/var/lib/kubelet/pods/37c04164-f683-4a4b-ad32-b0b7714172d5/volumes"
Sep 16 13:24:41.123640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:41.123604    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"]
Sep 16 13:24:41.124171 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:41.123824    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy" containerID="cri-o://f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27" gracePeriod=30
Sep 16 13:24:41.768519 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:41.768481    2577 patch_prober.go:28] interesting pod/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 13:24:41.768680 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:41.768541    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:24:46.400684 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.400621    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:24:46.540350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c9342-1c8e-43d2-8a7b-accae753f322-istio-podinfo\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.540350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540352    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-credential-socket\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.540589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-socket\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.540589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-istio-token\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.540589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540465    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-data\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.540589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-envoy\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.540589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-certs\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.540815 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:46.540815 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540665    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:46.540815 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540763    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-data" (OuterVolumeSpecName: "istio-data") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:46.541020 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c9342-1c8e-43d2-8a7b-accae753f322-istiod-ca-cert\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.541020 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:46.541020 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.540902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrsqv\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-kube-api-access-vrsqv\") pod \"960c9342-1c8e-43d2-8a7b-accae753f322\" (UID: \"960c9342-1c8e-43d2-8a7b-accae753f322\") "
Sep 16 13:24:46.541178 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.541152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960c9342-1c8e-43d2-8a7b-accae753f322-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:24:46.541494 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.541471    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.541580 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.541499    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.541580 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.541516    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.541580 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.541530    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.541580 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.541544    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960c9342-1c8e-43d2-8a7b-accae753f322-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.542765 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.542744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:46.542765 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.542753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/960c9342-1c8e-43d2-8a7b-accae753f322-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:24:46.542888 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.542823    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-istio-token" (OuterVolumeSpecName: "istio-token") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:46.542934 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.542883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-kube-api-access-vrsqv" (OuterVolumeSpecName: "kube-api-access-vrsqv") pod "960c9342-1c8e-43d2-8a7b-accae753f322" (UID: "960c9342-1c8e-43d2-8a7b-accae753f322"). InnerVolumeSpecName "kube-api-access-vrsqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:46.642915 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.642880    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.642915 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.642908    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960c9342-1c8e-43d2-8a7b-accae753f322-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.642915 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.642917    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrsqv\" (UniqueName: \"kubernetes.io/projected/960c9342-1c8e-43d2-8a7b-accae753f322-kube-api-access-vrsqv\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:46.643140 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:46.642930    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960c9342-1c8e-43d2-8a7b-accae753f322-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:24:47.240732 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.240691    2577 generic.go:356] "Generic (PLEG): container finished" podID="960c9342-1c8e-43d2-8a7b-accae753f322" containerID="f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27" exitCode=0
Sep 16 13:24:47.240950 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.240746    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"
Sep 16 13:24:47.240950 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.240786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" event={"ID":"960c9342-1c8e-43d2-8a7b-accae753f322","Type":"ContainerDied","Data":"f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27"}
Sep 16 13:24:47.240950 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.240879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp" event={"ID":"960c9342-1c8e-43d2-8a7b-accae753f322","Type":"ContainerDied","Data":"5ab725b6f2a2c13bc6655e83d667fe55e449b69e97c42cd3690f0847921b3826"}
Sep 16 13:24:47.240950 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.240911    2577 scope.go:117] "RemoveContainer" containerID="f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27"
Sep 16 13:24:47.252098 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.252075    2577 scope.go:117] "RemoveContainer" containerID="3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9"
Sep 16 13:24:47.264328 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.264301    2577 scope.go:117] "RemoveContainer" containerID="f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27"
Sep 16 13:24:47.264724 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:24:47.264695    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27\": container with ID starting with f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27 not found: ID does not exist" containerID="f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27"
Sep 16 13:24:47.264837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.264735    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27"} err="failed to get container status \"f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27\": rpc error: code = NotFound desc = could not find container \"f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27\": container with ID starting with f7bb07e675a8d7d3781a5f4273feb9bdecc62e70304ec9f422151321174b3f27 not found: ID does not exist"
Sep 16 13:24:47.264837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.264758    2577 scope.go:117] "RemoveContainer" containerID="3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9"
Sep 16 13:24:47.265151 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:24:47.265089    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9\": container with ID starting with 3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9 not found: ID does not exist" containerID="3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9"
Sep 16 13:24:47.265151 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.265125    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9"} err="failed to get container status \"3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9\": rpc error: code = NotFound desc = could not find container \"3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9\": container with ID starting with 3b41bc845cf626fbdad27a013712c5b1fe43a59f9831b12b7982f12e7280a9c9 not found: ID does not exist"
Sep 16 13:24:47.289801 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.289766    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"]
Sep 16 13:24:47.312540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:47.312505    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-15da7b-openshift-default-7d4b8d687d-r7gdp"]
Sep 16 13:24:49.111424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:49.111393    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" path="/var/lib/kubelet/pods/960c9342-1c8e-43d2-8a7b-accae753f322/volumes"
Sep 16 13:24:51.495738 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.495706    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"]
Sep 16 13:24:51.496193 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496180    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy"
Sep 16 13:24:51.496244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496194    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy"
Sep 16 13:24:51.496244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496204    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy"
Sep 16 13:24:51.496244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496209    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy"
Sep 16 13:24:51.496244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496220    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy"
Sep 16 13:24:51.496244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496226    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy"
Sep 16 13:24:51.496244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496232    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy"
Sep 16 13:24:51.496244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496236    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy"
Sep 16 13:24:51.496244 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496246    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217537df-4321-4e99-928a-c8f6ffe3320c" containerName="maas-api"
Sep 16 13:24:51.496487 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496252    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="217537df-4321-4e99-928a-c8f6ffe3320c" containerName="maas-api"
Sep 16 13:24:51.496487 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496321    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy"
Sep 16 13:24:51.496487 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496330    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="217537df-4321-4e99-928a-c8f6ffe3320c" containerName="maas-api"
Sep 16 13:24:51.496487 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496338    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="960c9342-1c8e-43d2-8a7b-accae753f322" containerName="istio-proxy"
Sep 16 13:24:51.496487 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496344    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy"
Sep 16 13:24:51.496487 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.496350    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37c04164-f683-4a4b-ad32-b0b7714172d5" containerName="istio-proxy"
Sep 16 13:24:51.499668 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.499650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.502285 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.502263    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-588a26-serving-cert\""
Sep 16 13:24:51.526888 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.526840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"]
Sep 16 13:24:51.593656 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.593616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed9bb09-a233-4804-bd99-6ac631f93013-openshift-service-ca\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.593837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.593697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-metrics-certs\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.593837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.593762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28f98\" (UniqueName: \"kubernetes.io/projected/aed9bb09-a233-4804-bd99-6ac631f93013-kube-api-access-28f98\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.593837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.593833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-maas-api-tls\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.695249 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.695212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed9bb09-a233-4804-bd99-6ac631f93013-openshift-service-ca\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.695446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.695266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-metrics-certs\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.695446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.695305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28f98\" (UniqueName: \"kubernetes.io/projected/aed9bb09-a233-4804-bd99-6ac631f93013-kube-api-access-28f98\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.695446 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.695348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-maas-api-tls\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.695935 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.695908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed9bb09-a233-4804-bd99-6ac631f93013-openshift-service-ca\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.697662 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.697640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-metrics-certs\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.697785 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.697689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-maas-api-tls\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.732168 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.732124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28f98\" (UniqueName: \"kubernetes.io/projected/aed9bb09-a233-4804-bd99-6ac631f93013-kube-api-access-28f98\") pod \"maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:51.809493 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:51.809387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:52.216670 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:52.216645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"]
Sep 16 13:24:52.219249 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:24:52.219223    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed9bb09_a233_4804_bd99_6ac631f93013.slice/crio-0012e364eb46a4747cc04a1d3a50d6d86eaae4664d99e036dc33cf739e7a78c3 WatchSource:0}: Error finding container 0012e364eb46a4747cc04a1d3a50d6d86eaae4664d99e036dc33cf739e7a78c3: Status 404 returned error can't find the container with id 0012e364eb46a4747cc04a1d3a50d6d86eaae4664d99e036dc33cf739e7a78c3
Sep 16 13:24:52.220681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:52.220660    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:24:52.270510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:52.270478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w" event={"ID":"aed9bb09-a233-4804-bd99-6ac631f93013","Type":"ContainerStarted","Data":"0012e364eb46a4747cc04a1d3a50d6d86eaae4664d99e036dc33cf739e7a78c3"}
Sep 16 13:24:53.278222 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:53.278186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w" event={"ID":"aed9bb09-a233-4804-bd99-6ac631f93013","Type":"ContainerStarted","Data":"7d3d07524e715a1ff3097c72fd89182e08f61cfc2ddab620337c5598a56e8ad6"}
Sep 16 13:24:53.278642 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:53.278302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:59.290305 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:59.290274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:24:59.329403 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:24:59.329348    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w" podStartSLOduration=7.780208389 podStartE2EDuration="8.32933357s" totalImagesPullingTime="549.125181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:53.314389478 +0000 UTC m=+964.737131058" watchObservedRunningTime="2026-09-16 13:24:59.32933357 +0000 UTC m=+970.752075149"
Sep 16 13:25:36.106676 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.106627    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"]
Sep 16 13:25:36.110667 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.110650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.114720 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.114692    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-588a26\"/\"kube-root-ca.crt\""
Sep 16 13:25:36.114886 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.114725    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-588a26\"/\"openshift-service-ca.crt\""
Sep 16 13:25:36.114886 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.114691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-588a26\"/\"explicit-model-588a26-kserve-self-signed-certs\""
Sep 16 13:25:36.114886 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.114691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-588a26\"/\"default-dockercfg-jzttc\""
Sep 16 13:25:36.124528 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.124505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"]
Sep 16 13:25:36.282093 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.282058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-dshm\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.282093 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.282097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-home\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.282359 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.282119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbxw\" (UniqueName: \"kubernetes.io/projected/b572aede-61c7-44d2-b72a-d8034607b1ac-kube-api-access-tsbxw\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.282359 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.282313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-tmp-dir\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.282486 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.282377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-model-cache\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.282486 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.282413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b572aede-61c7-44d2-b72a-d8034607b1ac-tls-certs\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384006 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.383918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-dshm\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384006 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.383960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-home\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384006 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.383976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbxw\" (UniqueName: \"kubernetes.io/projected/b572aede-61c7-44d2-b72a-d8034607b1ac-kube-api-access-tsbxw\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384284 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.384056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-tmp-dir\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384284 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.384095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-model-cache\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384284 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.384119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b572aede-61c7-44d2-b72a-d8034607b1ac-tls-certs\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384452 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.384378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-home\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384513 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.384487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-tmp-dir\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.384562 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.384534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-model-cache\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.386285 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.386266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-dshm\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.386556 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.386540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b572aede-61c7-44d2-b72a-d8034607b1ac-tls-certs\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.397139 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.397106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbxw\" (UniqueName: \"kubernetes.io/projected/b572aede-61c7-44d2-b72a-d8034607b1ac-kube-api-access-tsbxw\") pod \"explicit-model-588a26-kserve-689bd8b9d9-dv4df\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") " pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.420123 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.420103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:36.701762 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:36.701736    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"]
Sep 16 13:25:36.703701 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:25:36.703675    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb572aede_61c7_44d2_b72a_d8034607b1ac.slice/crio-5ab7b57ccf3e2805f21c4fd7d938fe5b4de926c2f90b44cfe147e56ace9c2f35 WatchSource:0}: Error finding container 5ab7b57ccf3e2805f21c4fd7d938fe5b4de926c2f90b44cfe147e56ace9c2f35: Status 404 returned error can't find the container with id 5ab7b57ccf3e2805f21c4fd7d938fe5b4de926c2f90b44cfe147e56ace9c2f35
Sep 16 13:25:37.530911 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:37.530841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df" event={"ID":"b572aede-61c7-44d2-b72a-d8034607b1ac","Type":"ContainerStarted","Data":"ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814"}
Sep 16 13:25:37.531318 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:37.530938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df" event={"ID":"b572aede-61c7-44d2-b72a-d8034607b1ac","Type":"ContainerStarted","Data":"5ab7b57ccf3e2805f21c4fd7d938fe5b4de926c2f90b44cfe147e56ace9c2f35"}
Sep 16 13:25:37.564444 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:37.564391    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df" podStartSLOduration=1.564375195 podStartE2EDuration="1.564375195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:25:37.562351741 +0000 UTC m=+1008.985093320" watchObservedRunningTime="2026-09-16 13:25:37.564375195 +0000 UTC m=+1008.987116773"
Sep 16 13:25:46.421039 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:46.420998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:46.421039 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:46.421055    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:46.434228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:46.434199    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:25:46.594157 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:25:46.594127    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:26:01.746890 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:01.746836    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"]
Sep 16 13:26:12.513708 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.513672    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"]
Sep 16 13:26:12.514151 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.513934    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w" podUID="aed9bb09-a233-4804-bd99-6ac631f93013" containerName="maas-api" containerID="cri-o://7d3d07524e715a1ff3097c72fd89182e08f61cfc2ddab620337c5598a56e8ad6" gracePeriod=30
Sep 16 13:26:12.739914 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.739843    2577 generic.go:356] "Generic (PLEG): container finished" podID="aed9bb09-a233-4804-bd99-6ac631f93013" containerID="7d3d07524e715a1ff3097c72fd89182e08f61cfc2ddab620337c5598a56e8ad6" exitCode=0
Sep 16 13:26:12.740074 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.739880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w" event={"ID":"aed9bb09-a233-4804-bd99-6ac631f93013","Type":"ContainerDied","Data":"7d3d07524e715a1ff3097c72fd89182e08f61cfc2ddab620337c5598a56e8ad6"}
Sep 16 13:26:12.772659 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.772598    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:26:12.881974 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.881942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-maas-api-tls\") pod \"aed9bb09-a233-4804-bd99-6ac631f93013\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") "
Sep 16 13:26:12.882158 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.882005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-metrics-certs\") pod \"aed9bb09-a233-4804-bd99-6ac631f93013\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") "
Sep 16 13:26:12.882158 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.882121    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28f98\" (UniqueName: \"kubernetes.io/projected/aed9bb09-a233-4804-bd99-6ac631f93013-kube-api-access-28f98\") pod \"aed9bb09-a233-4804-bd99-6ac631f93013\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") "
Sep 16 13:26:12.882303 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.882232    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed9bb09-a233-4804-bd99-6ac631f93013-openshift-service-ca\") pod \"aed9bb09-a233-4804-bd99-6ac631f93013\" (UID: \"aed9bb09-a233-4804-bd99-6ac631f93013\") "
Sep 16 13:26:12.882847 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.882763    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed9bb09-a233-4804-bd99-6ac631f93013-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aed9bb09-a233-4804-bd99-6ac631f93013" (UID: "aed9bb09-a233-4804-bd99-6ac631f93013"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:26:12.884622 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.884571    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aed9bb09-a233-4804-bd99-6ac631f93013" (UID: "aed9bb09-a233-4804-bd99-6ac631f93013"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:26:12.884842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.884808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aed9bb09-a233-4804-bd99-6ac631f93013" (UID: "aed9bb09-a233-4804-bd99-6ac631f93013"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:26:12.884977 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.884820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed9bb09-a233-4804-bd99-6ac631f93013-kube-api-access-28f98" (OuterVolumeSpecName: "kube-api-access-28f98") pod "aed9bb09-a233-4804-bd99-6ac631f93013" (UID: "aed9bb09-a233-4804-bd99-6ac631f93013"). InnerVolumeSpecName "kube-api-access-28f98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:26:12.983736 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.983698    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28f98\" (UniqueName: \"kubernetes.io/projected/aed9bb09-a233-4804-bd99-6ac631f93013-kube-api-access-28f98\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:12.983736 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.983731    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed9bb09-a233-4804-bd99-6ac631f93013-openshift-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:12.983736 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.983741    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-maas-api-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:12.984008 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:12.983752    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed9bb09-a233-4804-bd99-6ac631f93013-metrics-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:13.746617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:13.746588    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"
Sep 16 13:26:13.747070 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:13.746633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w" event={"ID":"aed9bb09-a233-4804-bd99-6ac631f93013","Type":"ContainerDied","Data":"0012e364eb46a4747cc04a1d3a50d6d86eaae4664d99e036dc33cf739e7a78c3"}
Sep 16 13:26:13.747070 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:13.746699    2577 scope.go:117] "RemoveContainer" containerID="7d3d07524e715a1ff3097c72fd89182e08f61cfc2ddab620337c5598a56e8ad6"
Sep 16 13:26:13.816282 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:13.816249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"]
Sep 16 13:26:13.826885 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:13.826843    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-588a26-84f96c6c6f-4vf9w"]
Sep 16 13:26:15.110100 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:15.110070    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed9bb09-a233-4804-bd99-6ac631f93013" path="/var/lib/kubelet/pods/aed9bb09-a233-4804-bd99-6ac631f93013/volumes"
Sep 16 13:26:16.766191 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:16.766137    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df" podUID="b572aede-61c7-44d2-b72a-d8034607b1ac" containerName="main" containerID="cri-o://ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814" gracePeriod=45
Sep 16 13:26:17.021598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.021533    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:26:17.123934 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.123904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b572aede-61c7-44d2-b72a-d8034607b1ac-tls-certs\") pod \"b572aede-61c7-44d2-b72a-d8034607b1ac\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") "
Sep 16 13:26:17.124117 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.123951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-home\") pod \"b572aede-61c7-44d2-b72a-d8034607b1ac\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") "
Sep 16 13:26:17.124117 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-tmp-dir\") pod \"b572aede-61c7-44d2-b72a-d8034607b1ac\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") "
Sep 16 13:26:17.124117 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-model-cache\") pod \"b572aede-61c7-44d2-b72a-d8034607b1ac\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") "
Sep 16 13:26:17.124291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsbxw\" (UniqueName: \"kubernetes.io/projected/b572aede-61c7-44d2-b72a-d8034607b1ac-kube-api-access-tsbxw\") pod \"b572aede-61c7-44d2-b72a-d8034607b1ac\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") "
Sep 16 13:26:17.124291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-dshm\") pod \"b572aede-61c7-44d2-b72a-d8034607b1ac\" (UID: \"b572aede-61c7-44d2-b72a-d8034607b1ac\") "
Sep 16 13:26:17.124291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-home" (OuterVolumeSpecName: "home") pod "b572aede-61c7-44d2-b72a-d8034607b1ac" (UID: "b572aede-61c7-44d2-b72a-d8034607b1ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:17.124291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124280    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b572aede-61c7-44d2-b72a-d8034607b1ac" (UID: "b572aede-61c7-44d2-b72a-d8034607b1ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:17.124481 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124400    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "b572aede-61c7-44d2-b72a-d8034607b1ac" (UID: "b572aede-61c7-44d2-b72a-d8034607b1ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:17.124582 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124565    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-home\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:17.124640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124587    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-tmp-dir\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:17.124640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.124600    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-model-cache\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:17.126218 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.126193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-dshm" (OuterVolumeSpecName: "dshm") pod "b572aede-61c7-44d2-b72a-d8034607b1ac" (UID: "b572aede-61c7-44d2-b72a-d8034607b1ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:17.126218 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.126193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b572aede-61c7-44d2-b72a-d8034607b1ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b572aede-61c7-44d2-b72a-d8034607b1ac" (UID: "b572aede-61c7-44d2-b72a-d8034607b1ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:26:17.126411 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.126221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b572aede-61c7-44d2-b72a-d8034607b1ac-kube-api-access-tsbxw" (OuterVolumeSpecName: "kube-api-access-tsbxw") pod "b572aede-61c7-44d2-b72a-d8034607b1ac" (UID: "b572aede-61c7-44d2-b72a-d8034607b1ac"). InnerVolumeSpecName "kube-api-access-tsbxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:26:17.226153 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.226116    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsbxw\" (UniqueName: \"kubernetes.io/projected/b572aede-61c7-44d2-b72a-d8034607b1ac-kube-api-access-tsbxw\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:17.226153 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.226146    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b572aede-61c7-44d2-b72a-d8034607b1ac-dshm\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:17.226153 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.226155    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b572aede-61c7-44d2-b72a-d8034607b1ac-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:26:17.772363 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.772323    2577 generic.go:356] "Generic (PLEG): container finished" podID="b572aede-61c7-44d2-b72a-d8034607b1ac" containerID="ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814" exitCode=0
Sep 16 13:26:17.772821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.772369    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"
Sep 16 13:26:17.772821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.772393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df" event={"ID":"b572aede-61c7-44d2-b72a-d8034607b1ac","Type":"ContainerDied","Data":"ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814"}
Sep 16 13:26:17.772821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.772435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df" event={"ID":"b572aede-61c7-44d2-b72a-d8034607b1ac","Type":"ContainerDied","Data":"5ab7b57ccf3e2805f21c4fd7d938fe5b4de926c2f90b44cfe147e56ace9c2f35"}
Sep 16 13:26:17.772821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.772457    2577 scope.go:117] "RemoveContainer" containerID="ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814"
Sep 16 13:26:17.786127 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.786107    2577 scope.go:117] "RemoveContainer" containerID="ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814"
Sep 16 13:26:17.786439 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:26:17.786420    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814\": container with ID starting with ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814 not found: ID does not exist" containerID="ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814"
Sep 16 13:26:17.786498 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.786454    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814"} err="failed to get container status \"ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814\": rpc error: code = NotFound desc = could not find container \"ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814\": container with ID starting with ecbb92860036464140abaf6cca5c8874c997237bc47155f6eb5a3283f056b814 not found: ID does not exist"
Sep 16 13:26:17.803776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.803746    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"]
Sep 16 13:26:17.808447 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:17.808425    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-588a26/explicit-model-588a26-kserve-689bd8b9d9-dv4df"]
Sep 16 13:26:19.110264 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:19.110228    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b572aede-61c7-44d2-b72a-d8034607b1ac" path="/var/lib/kubelet/pods/b572aede-61c7-44d2-b72a-d8034607b1ac/volumes"
Sep 16 13:26:20.666203 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.666165    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"]
Sep 16 13:26:20.666714 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.666697    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed9bb09-a233-4804-bd99-6ac631f93013" containerName="maas-api"
Sep 16 13:26:20.666755 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.666716    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed9bb09-a233-4804-bd99-6ac631f93013" containerName="maas-api"
Sep 16 13:26:20.666755 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.666739    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b572aede-61c7-44d2-b72a-d8034607b1ac" containerName="main"
Sep 16 13:26:20.666755 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.666744    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b572aede-61c7-44d2-b72a-d8034607b1ac" containerName="main"
Sep 16 13:26:20.666900 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.666814    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed9bb09-a233-4804-bd99-6ac631f93013" containerName="maas-api"
Sep 16 13:26:20.666900 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.666823    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b572aede-61c7-44d2-b72a-d8034607b1ac" containerName="main"
Sep 16 13:26:20.671435 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.671397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.673949 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.673930    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-3ec3e3-openshift-default-dockercfg-8zxzg\""
Sep 16 13:26:20.689611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.689583    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"]
Sep 16 13:26:20.762417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.762352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-envoy\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.762669 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.762560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-certs\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.762669 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.762633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-istio-token\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.762800 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.762669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35873993-b4e4-446d-a50c-5bbe963b028e-istiod-ca-cert\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.762800 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.762833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wct5s\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-kube-api-access-wct5s\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.763107 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.762981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-data\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.763107 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.763070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-socket\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.763453 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.763172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-credential-socket\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.763453 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.763236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35873993-b4e4-446d-a50c-5bbe963b028e-istio-podinfo\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864130 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wct5s\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-kube-api-access-wct5s\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-data\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-socket\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-credential-socket\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35873993-b4e4-446d-a50c-5bbe963b028e-istio-podinfo\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-envoy\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-certs\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-istio-token\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864693 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35873993-b4e4-446d-a50c-5bbe963b028e-istiod-ca-cert\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864693 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-data\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864693 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-socket\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.864956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.864934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-credential-socket\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.865048 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.865032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-certs\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.865410 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.865393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35873993-b4e4-446d-a50c-5bbe963b028e-istiod-ca-cert\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.866637 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.866608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-envoy\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.866747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.866727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35873993-b4e4-446d-a50c-5bbe963b028e-istio-podinfo\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.877245 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.877220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-istio-token\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.879321 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.879292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wct5s\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-kube-api-access-wct5s\") pod \"e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") " pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:20.982237 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:20.982203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:21.271090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:21.271059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"]
Sep 16 13:26:21.273436 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:26:21.273407    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35873993_b4e4_446d_a50c_5bbe963b028e.slice/crio-ec5ba15c05fd8ec9ed8bd2c5abdeffc015087b2db11d6e3f0a5d75b6ac3acdf4 WatchSource:0}: Error finding container ec5ba15c05fd8ec9ed8bd2c5abdeffc015087b2db11d6e3f0a5d75b6ac3acdf4: Status 404 returned error can't find the container with id ec5ba15c05fd8ec9ed8bd2c5abdeffc015087b2db11d6e3f0a5d75b6ac3acdf4
Sep 16 13:26:21.276003 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:21.275957    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:26:21.276126 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:21.276086    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:26:21.276188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:21.276150    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:26:21.800052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:21.800015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d" event={"ID":"35873993-b4e4-446d-a50c-5bbe963b028e","Type":"ContainerStarted","Data":"9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f"}
Sep 16 13:26:21.800530 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:21.800078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d" event={"ID":"35873993-b4e4-446d-a50c-5bbe963b028e","Type":"ContainerStarted","Data":"ec5ba15c05fd8ec9ed8bd2c5abdeffc015087b2db11d6e3f0a5d75b6ac3acdf4"}
Sep 16 13:26:21.845194 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:21.845131    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d" podStartSLOduration=1.845112888 podStartE2EDuration="1.845112888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:26:21.842995578 +0000 UTC m=+1053.265737157" watchObservedRunningTime="2026-09-16 13:26:21.845112888 +0000 UTC m=+1053.267854467"
Sep 16 13:26:21.983328 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:21.983293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:22.988828 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:22.988794    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:23.813301 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:23.813266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:23.814050 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:23.814031    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:26:54.876250 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.876216    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"]
Sep 16 13:26:54.881597 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.881570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:54.884079 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.884051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3ec3e3\"/\"openshift-service-ca.crt\""
Sep 16 13:26:54.884758 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.884737    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3ec3e3\"/\"default-dockercfg-vq72p\""
Sep 16 13:26:54.884758 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.884753    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3ec3e3\"/\"kube-root-ca.crt\""
Sep 16 13:26:54.884942 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.884823    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3ec3e3\"/\"noten-model-3ec3e3-kserve-self-signed-certs\""
Sep 16 13:26:54.890560 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.890538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"]
Sep 16 13:26:54.918154 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.918122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-tmp-dir\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:54.918310 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.918174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40691aab-ce18-4007-ba04-4bb8e09d82b2-tls-certs\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:54.918310 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.918268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-model-cache\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:54.918310 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.918306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-home\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:54.918419 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.918325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjqln\" (UniqueName: \"kubernetes.io/projected/40691aab-ce18-4007-ba04-4bb8e09d82b2-kube-api-access-qjqln\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:54.918419 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:54.918387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-dshm\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.019555 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.019523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-tmp-dir\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.019747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.019579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40691aab-ce18-4007-ba04-4bb8e09d82b2-tls-certs\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.019747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.019630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-model-cache\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.019747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.019656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-home\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.019747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.019675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjqln\" (UniqueName: \"kubernetes.io/projected/40691aab-ce18-4007-ba04-4bb8e09d82b2-kube-api-access-qjqln\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.019747 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.019712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-dshm\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.020055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.019974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-tmp-dir\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.020055 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.020019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-model-cache\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.020155 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.020122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-home\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.022080 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.022059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-dshm\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.022227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.022209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40691aab-ce18-4007-ba04-4bb8e09d82b2-tls-certs\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.028951 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.028928    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjqln\" (UniqueName: \"kubernetes.io/projected/40691aab-ce18-4007-ba04-4bb8e09d82b2-kube-api-access-qjqln\") pod \"noten-model-3ec3e3-kserve-7d957fdc55-bfrqv\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") " pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.193707 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.193677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:26:55.466402 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.466372    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"]
Sep 16 13:26:55.467805 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:26:55.467780    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40691aab_ce18_4007_ba04_4bb8e09d82b2.slice/crio-5ae9870dd112c35e849bbb4a8f44cc23761a3dc03b8fd4b8f8fb9f1f027d9450 WatchSource:0}: Error finding container 5ae9870dd112c35e849bbb4a8f44cc23761a3dc03b8fd4b8f8fb9f1f027d9450: Status 404 returned error can't find the container with id 5ae9870dd112c35e849bbb4a8f44cc23761a3dc03b8fd4b8f8fb9f1f027d9450
Sep 16 13:26:55.993072 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.993032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv" event={"ID":"40691aab-ce18-4007-ba04-4bb8e09d82b2","Type":"ContainerStarted","Data":"3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b"}
Sep 16 13:26:55.993458 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:26:55.993084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv" event={"ID":"40691aab-ce18-4007-ba04-4bb8e09d82b2","Type":"ContainerStarted","Data":"5ae9870dd112c35e849bbb4a8f44cc23761a3dc03b8fd4b8f8fb9f1f027d9450"}
Sep 16 13:27:05.194023 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:05.193983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:27:05.194568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:05.194036    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:27:05.206636 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:05.206612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:27:05.226383 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:05.226328    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv" podStartSLOduration=11.226312502 podStartE2EDuration="11.226312502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:26:56.016695115 +0000 UTC m=+1087.439436694" watchObservedRunningTime="2026-09-16 13:27:05.226312502 +0000 UTC m=+1096.649054082"
Sep 16 13:27:06.062198 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:06.062130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:27:11.333309 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:11.333267    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"]
Sep 16 13:27:11.446168 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:11.446133    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"]
Sep 16 13:27:11.446416 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:11.446370    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d" podUID="35873993-b4e4-446d-a50c-5bbe963b028e" containerName="istio-proxy" containerID="cri-o://9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f" gracePeriod=30
Sep 16 13:27:16.701457 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.701433    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:27:16.848091 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.847996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-certs\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848091 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wct5s\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-kube-api-access-wct5s\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848325 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-credential-socket\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848325 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848137    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-envoy\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848325 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848192    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35873993-b4e4-446d-a50c-5bbe963b028e-istio-podinfo\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848325 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-data\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848325 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848269    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-socket\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848576 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:16.848576 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848366    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-istio-token\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848576 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35873993-b4e4-446d-a50c-5bbe963b028e-istiod-ca-cert\") pod \"35873993-b4e4-446d-a50c-5bbe963b028e\" (UID: \"35873993-b4e4-446d-a50c-5bbe963b028e\") "
Sep 16 13:27:16.848576 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:16.848763 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:16.848845 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848817    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:16.848845 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848845    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:16.849067 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848876    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:16.849067 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-data" (OuterVolumeSpecName: "istio-data") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:16.849067 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.848943    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35873993-b4e4-446d-a50c-5bbe963b028e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:27:16.850408 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.850382    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:16.850534 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.850433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-kube-api-access-wct5s" (OuterVolumeSpecName: "kube-api-access-wct5s") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "kube-api-access-wct5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:27:16.850802 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.850779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35873993-b4e4-446d-a50c-5bbe963b028e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:27:16.850893 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.850794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-istio-token" (OuterVolumeSpecName: "istio-token") pod "35873993-b4e4-446d-a50c-5bbe963b028e" (UID: "35873993-b4e4-446d-a50c-5bbe963b028e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:27:16.950382 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.950349    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35873993-b4e4-446d-a50c-5bbe963b028e-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:16.950382 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.950378    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:16.950382 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.950388    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:16.950610 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.950395    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35873993-b4e4-446d-a50c-5bbe963b028e-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:16.950610 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.950403    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wct5s\" (UniqueName: \"kubernetes.io/projected/35873993-b4e4-446d-a50c-5bbe963b028e-kube-api-access-wct5s\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:16.950610 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:16.950412    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35873993-b4e4-446d-a50c-5bbe963b028e-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:17.123742 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.123641    2577 generic.go:356] "Generic (PLEG): container finished" podID="35873993-b4e4-446d-a50c-5bbe963b028e" containerID="9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f" exitCode=0
Sep 16 13:27:17.123742 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.123695    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"
Sep 16 13:27:17.123742 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.123723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d" event={"ID":"35873993-b4e4-446d-a50c-5bbe963b028e","Type":"ContainerDied","Data":"9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f"}
Sep 16 13:27:17.124022 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.123774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d" event={"ID":"35873993-b4e4-446d-a50c-5bbe963b028e","Type":"ContainerDied","Data":"ec5ba15c05fd8ec9ed8bd2c5abdeffc015087b2db11d6e3f0a5d75b6ac3acdf4"}
Sep 16 13:27:17.124022 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.123796    2577 scope.go:117] "RemoveContainer" containerID="9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f"
Sep 16 13:27:17.136263 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.136239    2577 scope.go:117] "RemoveContainer" containerID="9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f"
Sep 16 13:27:17.136623 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:27:17.136595    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f\": container with ID starting with 9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f not found: ID does not exist" containerID="9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f"
Sep 16 13:27:17.136722 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.136633    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f"} err="failed to get container status \"9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f\": rpc error: code = NotFound desc = could not find container \"9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f\": container with ID starting with 9e8a639db7ca4eaaa5ca146a7cbe69327ee2a5f6fba2a13323895c3730dd330f not found: ID does not exist"
Sep 16 13:27:17.155595 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.155562    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"]
Sep 16 13:27:17.159729 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:17.159698    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-3ec3e3-openshift-default-65c696cf4d-x9v2d"]
Sep 16 13:27:19.111023 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:19.110971    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35873993-b4e4-446d-a50c-5bbe963b028e" path="/var/lib/kubelet/pods/35873993-b4e4-446d-a50c-5bbe963b028e/volumes"
Sep 16 13:27:26.355344 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.355302    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv" podUID="40691aab-ce18-4007-ba04-4bb8e09d82b2" containerName="main" containerID="cri-o://3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b" gracePeriod=45
Sep 16 13:27:26.615373 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.615309    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:27:26.653342 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653305    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-tmp-dir\") pod \"40691aab-ce18-4007-ba04-4bb8e09d82b2\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") "
Sep 16 13:27:26.653510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-dshm\") pod \"40691aab-ce18-4007-ba04-4bb8e09d82b2\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") "
Sep 16 13:27:26.653510 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjqln\" (UniqueName: \"kubernetes.io/projected/40691aab-ce18-4007-ba04-4bb8e09d82b2-kube-api-access-qjqln\") pod \"40691aab-ce18-4007-ba04-4bb8e09d82b2\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") "
Sep 16 13:27:26.653618 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40691aab-ce18-4007-ba04-4bb8e09d82b2-tls-certs\") pod \"40691aab-ce18-4007-ba04-4bb8e09d82b2\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") "
Sep 16 13:27:26.653618 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653566    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-home\") pod \"40691aab-ce18-4007-ba04-4bb8e09d82b2\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") "
Sep 16 13:27:26.653618 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653585    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40691aab-ce18-4007-ba04-4bb8e09d82b2" (UID: "40691aab-ce18-4007-ba04-4bb8e09d82b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:26.653618 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-model-cache\") pod \"40691aab-ce18-4007-ba04-4bb8e09d82b2\" (UID: \"40691aab-ce18-4007-ba04-4bb8e09d82b2\") "
Sep 16 13:27:26.653827 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "40691aab-ce18-4007-ba04-4bb8e09d82b2" (UID: "40691aab-ce18-4007-ba04-4bb8e09d82b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:26.653978 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.653957    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-home" (OuterVolumeSpecName: "home") pod "40691aab-ce18-4007-ba04-4bb8e09d82b2" (UID: "40691aab-ce18-4007-ba04-4bb8e09d82b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:26.654180 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.654160    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-home\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:26.654234 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.654186    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-model-cache\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:26.654234 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.654201    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-tmp-dir\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:26.655660 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.655633    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-dshm" (OuterVolumeSpecName: "dshm") pod "40691aab-ce18-4007-ba04-4bb8e09d82b2" (UID: "40691aab-ce18-4007-ba04-4bb8e09d82b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:26.655921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.655899    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40691aab-ce18-4007-ba04-4bb8e09d82b2-kube-api-access-qjqln" (OuterVolumeSpecName: "kube-api-access-qjqln") pod "40691aab-ce18-4007-ba04-4bb8e09d82b2" (UID: "40691aab-ce18-4007-ba04-4bb8e09d82b2"). InnerVolumeSpecName "kube-api-access-qjqln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:27:26.656000 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.655920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40691aab-ce18-4007-ba04-4bb8e09d82b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40691aab-ce18-4007-ba04-4bb8e09d82b2" (UID: "40691aab-ce18-4007-ba04-4bb8e09d82b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:27:26.755516 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.755477    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjqln\" (UniqueName: \"kubernetes.io/projected/40691aab-ce18-4007-ba04-4bb8e09d82b2-kube-api-access-qjqln\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:26.755516 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.755505    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40691aab-ce18-4007-ba04-4bb8e09d82b2-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:26.755516 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:26.755514    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40691aab-ce18-4007-ba04-4bb8e09d82b2-dshm\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:27:27.182876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.182820    2577 generic.go:356] "Generic (PLEG): container finished" podID="40691aab-ce18-4007-ba04-4bb8e09d82b2" containerID="3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b" exitCode=0
Sep 16 13:27:27.182876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.182877    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"
Sep 16 13:27:27.183090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.182901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv" event={"ID":"40691aab-ce18-4007-ba04-4bb8e09d82b2","Type":"ContainerDied","Data":"3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b"}
Sep 16 13:27:27.183090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.182944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv" event={"ID":"40691aab-ce18-4007-ba04-4bb8e09d82b2","Type":"ContainerDied","Data":"5ae9870dd112c35e849bbb4a8f44cc23761a3dc03b8fd4b8f8fb9f1f027d9450"}
Sep 16 13:27:27.183090 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.182963    2577 scope.go:117] "RemoveContainer" containerID="3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b"
Sep 16 13:27:27.191932 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.191913    2577 scope.go:117] "RemoveContainer" containerID="3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b"
Sep 16 13:27:27.192221 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:27:27.192202    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b\": container with ID starting with 3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b not found: ID does not exist" containerID="3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b"
Sep 16 13:27:27.192274 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.192230    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b"} err="failed to get container status \"3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b\": rpc error: code = NotFound desc = could not find container \"3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b\": container with ID starting with 3c9006f19a2a06efe6708e4b72e798af9aa8327be5262bd05421717dc0a62b3b not found: ID does not exist"
Sep 16 13:27:27.199834 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.199808    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"]
Sep 16 13:27:27.202087 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:27.202063    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-3ec3e3/noten-model-3ec3e3-kserve-7d957fdc55-bfrqv"]
Sep 16 13:27:28.213684 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.213648    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"]
Sep 16 13:27:28.214387 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.214370    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40691aab-ce18-4007-ba04-4bb8e09d82b2" containerName="main"
Sep 16 13:27:28.214466 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.214390    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="40691aab-ce18-4007-ba04-4bb8e09d82b2" containerName="main"
Sep 16 13:27:28.214466 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.214440    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35873993-b4e4-446d-a50c-5bbe963b028e" containerName="istio-proxy"
Sep 16 13:27:28.214466 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.214449    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35873993-b4e4-446d-a50c-5bbe963b028e" containerName="istio-proxy"
Sep 16 13:27:28.214568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.214552    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="40691aab-ce18-4007-ba04-4bb8e09d82b2" containerName="main"
Sep 16 13:27:28.214568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.214566    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35873993-b4e4-446d-a50c-5bbe963b028e" containerName="istio-proxy"
Sep 16 13:27:28.219688 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.219660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.223476 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.223179    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"distinct-w4-f27ea8-kserve-self-signed-certs\""
Sep 16 13:27:28.223476 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.223245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"openshift-service-ca.crt\""
Sep 16 13:27:28.223476 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.223388    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"kube-root-ca.crt\""
Sep 16 13:27:28.223948 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.223926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"default-dockercfg-2ckdl\""
Sep 16 13:27:28.226569 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.226544    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"]
Sep 16 13:27:28.272679 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.272643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-home\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.272679 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.272683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-tmp-dir\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.272931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.272775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-model-cache\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.272931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.272838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b24411-5fec-4d9c-a930-d659640ae0d1-tls-certs\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.272931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.272869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjcnc\" (UniqueName: \"kubernetes.io/projected/65b24411-5fec-4d9c-a930-d659640ae0d1-kube-api-access-qjcnc\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.272931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.272897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-dshm\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374032 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.373990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-model-cache\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374225 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.374079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b24411-5fec-4d9c-a930-d659640ae0d1-tls-certs\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374225 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.374113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcnc\" (UniqueName: \"kubernetes.io/projected/65b24411-5fec-4d9c-a930-d659640ae0d1-kube-api-access-qjcnc\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374225 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.374149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-dshm\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374225 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.374200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-home\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374225 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.374223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-tmp-dir\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374512 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.374489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-model-cache\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374589 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.374528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-tmp-dir\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.374726 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.374702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-home\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.377531 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.377504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-dshm\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.377678 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.377568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b24411-5fec-4d9c-a930-d659640ae0d1-tls-certs\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.384803 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.384775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjcnc\" (UniqueName: \"kubernetes.io/projected/65b24411-5fec-4d9c-a930-d659640ae0d1-kube-api-access-qjcnc\") pod \"distinct-w4-f27ea8-kserve-854596d86b-6tr6j\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.532007 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.531910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:28.809944 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:28.809867    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"]
Sep 16 13:27:28.813389 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:27:28.813360    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65b24411_5fec_4d9c_a930_d659640ae0d1.slice/crio-d569a519f8f599bb7090b66d8375ef77037a4a6a14944f7c9f78e451e3d9038d WatchSource:0}: Error finding container d569a519f8f599bb7090b66d8375ef77037a4a6a14944f7c9f78e451e3d9038d: Status 404 returned error can't find the container with id d569a519f8f599bb7090b66d8375ef77037a4a6a14944f7c9f78e451e3d9038d
Sep 16 13:27:29.111061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:29.110967    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40691aab-ce18-4007-ba04-4bb8e09d82b2" path="/var/lib/kubelet/pods/40691aab-ce18-4007-ba04-4bb8e09d82b2/volumes"
Sep 16 13:27:29.198003 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:29.197971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j" event={"ID":"65b24411-5fec-4d9c-a930-d659640ae0d1","Type":"ContainerStarted","Data":"72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b"}
Sep 16 13:27:29.198003 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:29.198007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j" event={"ID":"65b24411-5fec-4d9c-a930-d659640ae0d1","Type":"ContainerStarted","Data":"d569a519f8f599bb7090b66d8375ef77037a4a6a14944f7c9f78e451e3d9038d"}
Sep 16 13:27:29.216201 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:29.216151    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j" podStartSLOduration=1.216134224 podStartE2EDuration="1.216134224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:27:29.215064559 +0000 UTC m=+1120.637806140" watchObservedRunningTime="2026-09-16 13:27:29.216134224 +0000 UTC m=+1120.638875802"
Sep 16 13:27:38.532941 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:38.532910    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:38.533561 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:38.533005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:38.545560 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:38.545534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:27:39.262904 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:27:39.262877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:29:39.018178 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.018139    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"]
Sep 16 13:29:39.021732 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.021715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.024363 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.024336    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b97e79df-serving-cert\""
Sep 16 13:29:39.047334 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.047296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"]
Sep 16 13:29:39.064096 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.064060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-maas-api-tls\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.064253 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.064125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-openshift-service-ca\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.064253 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.064174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-metrics-certs\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.064253 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.064197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2kk\" (UniqueName: \"kubernetes.io/projected/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-kube-api-access-kk2kk\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.164910 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.164838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-maas-api-tls\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.165114 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.164971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-openshift-service-ca\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.165114 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.165054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-metrics-certs\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.165114 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.165083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2kk\" (UniqueName: \"kubernetes.io/projected/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-kube-api-access-kk2kk\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.165977 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.165952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-openshift-service-ca\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.167426 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.167394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-maas-api-tls\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.167525 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.167448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-metrics-certs\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.174111 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.174084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2kk\" (UniqueName: \"kubernetes.io/projected/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-kube-api-access-kk2kk\") pod \"maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.332250 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.332143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:39.615943 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.615913    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"]
Sep 16 13:29:39.617838 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:29:39.617813    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6787a2c_59aa_40d6_8d42_0ebf74e9ea09.slice/crio-529dd4a2b75208f045f44c208f24da25a356aff976ff3797eb9b1466a7dc8c47 WatchSource:0}: Error finding container 529dd4a2b75208f045f44c208f24da25a356aff976ff3797eb9b1466a7dc8c47: Status 404 returned error can't find the container with id 529dd4a2b75208f045f44c208f24da25a356aff976ff3797eb9b1466a7dc8c47
Sep 16 13:29:39.951176 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:39.951140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd" event={"ID":"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09","Type":"ContainerStarted","Data":"529dd4a2b75208f045f44c208f24da25a356aff976ff3797eb9b1466a7dc8c47"}
Sep 16 13:29:40.958031 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:40.957996    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd" event={"ID":"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09","Type":"ContainerStarted","Data":"7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae"}
Sep 16 13:29:40.958417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:40.958061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:29:40.978903 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:40.978838    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd" podStartSLOduration=2.506243316 podStartE2EDuration="2.97882193s" totalImagesPullingTime="472.578614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:29:40.976807343 +0000 UTC m=+1252.399548943" watchObservedRunningTime="2026-09-16 13:29:40.97882193 +0000 UTC m=+1252.401563508"
Sep 16 13:29:46.969338 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:29:46.969306    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:30:19.274369 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.274335    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"]
Sep 16 13:30:19.275031 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.274601    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd" podUID="f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" containerName="maas-api" containerID="cri-o://7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae" gracePeriod=30
Sep 16 13:30:19.545183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.545156    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:30:19.611218 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.611186    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-maas-api-tls\") pod \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") "
Sep 16 13:30:19.611413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.611310    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-metrics-certs\") pod \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") "
Sep 16 13:30:19.611413 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.611343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk2kk\" (UniqueName: \"kubernetes.io/projected/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-kube-api-access-kk2kk\") pod \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") "
Sep 16 13:30:19.611537 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.611446    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-openshift-service-ca\") pod \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\" (UID: \"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09\") "
Sep 16 13:30:19.611875 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.611824    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" (UID: "f6787a2c-59aa-40d6-8d42-0ebf74e9ea09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:30:19.613417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.613387    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-kube-api-access-kk2kk" (OuterVolumeSpecName: "kube-api-access-kk2kk") pod "f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" (UID: "f6787a2c-59aa-40d6-8d42-0ebf74e9ea09"). InnerVolumeSpecName "kube-api-access-kk2kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:30:19.613535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.613447    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" (UID: "f6787a2c-59aa-40d6-8d42-0ebf74e9ea09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:30:19.613535 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.613480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" (UID: "f6787a2c-59aa-40d6-8d42-0ebf74e9ea09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:30:19.713064 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.713026    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-maas-api-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:30:19.713064 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.713063    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-metrics-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:30:19.713293 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.713077    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk2kk\" (UniqueName: \"kubernetes.io/projected/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-kube-api-access-kk2kk\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:30:19.713293 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:19.713092    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09-openshift-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:30:20.198825 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.198786    2577 generic.go:356] "Generic (PLEG): container finished" podID="f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" containerID="7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae" exitCode=0
Sep 16 13:30:20.199097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.198835    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"
Sep 16 13:30:20.199097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.198885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd" event={"ID":"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09","Type":"ContainerDied","Data":"7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae"}
Sep 16 13:30:20.199097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.198946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd" event={"ID":"f6787a2c-59aa-40d6-8d42-0ebf74e9ea09","Type":"ContainerDied","Data":"529dd4a2b75208f045f44c208f24da25a356aff976ff3797eb9b1466a7dc8c47"}
Sep 16 13:30:20.199097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.198969    2577 scope.go:117] "RemoveContainer" containerID="7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae"
Sep 16 13:30:20.212522 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.212498    2577 scope.go:117] "RemoveContainer" containerID="7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae"
Sep 16 13:30:20.212817 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:30:20.212796    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae\": container with ID starting with 7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae not found: ID does not exist" containerID="7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae"
Sep 16 13:30:20.212894 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.212827    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae"} err="failed to get container status \"7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae\": rpc error: code = NotFound desc = could not find container \"7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae\": container with ID starting with 7c286236e24027a38c22d153ae5d6606eeb8bdbb021a31b4b438c7b99f10f5ae not found: ID does not exist"
Sep 16 13:30:20.227272 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.227242    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"]
Sep 16 13:30:20.231239 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:20.231213    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b97e79df-854f895c5f-2s9sd"]
Sep 16 13:30:21.111098 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:21.111061    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" path="/var/lib/kubelet/pods/f6787a2c-59aa-40d6-8d42-0ebf74e9ea09/volumes"
Sep 16 13:30:45.335668 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.335636    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"]
Sep 16 13:30:45.336212 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.336197    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" containerName="maas-api"
Sep 16 13:30:45.336274 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.336214    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" containerName="maas-api"
Sep 16 13:30:45.336312 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.336302    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6787a2c-59aa-40d6-8d42-0ebf74e9ea09" containerName="maas-api"
Sep 16 13:30:45.341174 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.341151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.343895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.343875    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b0514e-serving-cert\""
Sep 16 13:30:45.350720 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.350693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"]
Sep 16 13:30:45.383949 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.383918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445f919-a104-41fc-ae2e-cdc67a0de187-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.384097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.383955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4hs\" (UniqueName: \"kubernetes.io/projected/9445f919-a104-41fc-ae2e-cdc67a0de187-kube-api-access-hn4hs\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.384097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.384057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.384097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.384083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.485297 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.485260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445f919-a104-41fc-ae2e-cdc67a0de187-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.485297 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.485298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4hs\" (UniqueName: \"kubernetes.io/projected/9445f919-a104-41fc-ae2e-cdc67a0de187-kube-api-access-hn4hs\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.485522 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.485344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.485522 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.485360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.485522 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:30:45.485502    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-serving-cert: secret "maas-api-e2e-shared-a-w2-b0514e-serving-cert" not found
Sep 16 13:30:45.485640 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:30:45.485582    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls podName:9445f919-a104-41fc-ae2e-cdc67a0de187 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:45.98556297 +0000 UTC m=+1317.408304531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls") pod "maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j" (UID: "9445f919-a104-41fc-ae2e-cdc67a0de187") : secret "maas-api-e2e-shared-a-w2-b0514e-serving-cert" not found
Sep 16 13:30:45.486002 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.485979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445f919-a104-41fc-ae2e-cdc67a0de187-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.487813 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.487791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.497945 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.497922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4hs\" (UniqueName: \"kubernetes.io/projected/9445f919-a104-41fc-ae2e-cdc67a0de187-kube-api-access-hn4hs\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.990482 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.990437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:45.992979 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:45.992954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:46.252417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:46.252324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:46.535706 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:46.535667    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"]
Sep 16 13:30:46.537570 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:30:46.537534    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9445f919_a104_41fc_ae2e_cdc67a0de187.slice/crio-94703875fe1dadfb428ca7ce386ab4152923690f609fd62d78d51bef6f507fe3 WatchSource:0}: Error finding container 94703875fe1dadfb428ca7ce386ab4152923690f609fd62d78d51bef6f507fe3: Status 404 returned error can't find the container with id 94703875fe1dadfb428ca7ce386ab4152923690f609fd62d78d51bef6f507fe3
Sep 16 13:30:46.538953 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:46.538936    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:30:47.350502 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:47.350396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j" event={"ID":"9445f919-a104-41fc-ae2e-cdc67a0de187","Type":"ContainerStarted","Data":"2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23"}
Sep 16 13:30:47.350502 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:47.350455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j" event={"ID":"9445f919-a104-41fc-ae2e-cdc67a0de187","Type":"ContainerStarted","Data":"94703875fe1dadfb428ca7ce386ab4152923690f609fd62d78d51bef6f507fe3"}
Sep 16 13:30:47.350724 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:47.350571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:53.361983 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:53.361952    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:30:53.383630 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:30:53.383579    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j" podStartSLOduration=7.969442954 podStartE2EDuration="8.383563993s" totalImagesPullingTime="414.121039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:47.371360767 +0000 UTC m=+1318.794102357" watchObservedRunningTime="2026-09-16 13:30:53.383563993 +0000 UTC m=+1324.806305614"
Sep 16 13:31:08.371258 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.371220    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"]
Sep 16 13:31:08.376318 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.376293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.379124 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.379100    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7841d9-openshift-default-dockercfg-lbd4l\""
Sep 16 13:31:08.392702 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.392674    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"]
Sep 16 13:31:08.432919 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.432847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istiod-ca-cert\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.432919 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.432916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-token\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.433183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.433022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-data\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.433183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.433071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-certs\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.433183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.433107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-socket\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.433183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.433142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55fd\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-kube-api-access-v55fd\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.433337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.433192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-podinfo\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.433337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.433227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-credential-socket\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.433337 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.433246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-envoy\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istiod-ca-cert\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534330 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-token\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534400 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-data\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534400 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-certs\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534525 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-socket\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534525 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55fd\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-kube-api-access-v55fd\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534525 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-podinfo\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534658 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-credential-socket\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534658 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-envoy\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534804 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-certs\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534876 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-socket\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.534965 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.534940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istiod-ca-cert\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.535231 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.535203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-credential-socket\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.535336 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.535281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-data\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.536882 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.536860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-envoy\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.537250 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.537228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-podinfo\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.545273 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.545246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-token\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.545516 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.545499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55fd\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-kube-api-access-v55fd\") pod \"e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") " pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.687100 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.687018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:08.968715 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.968686    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"]
Sep 16 13:31:08.970701 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:31:08.970668    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac20dd4_6c19_4382_84b8_bfb05c74ae58.slice/crio-8ff090048f5d3819a9fa39d1941bf7fe3079aad69fe8b34409ace443a8db92c1 WatchSource:0}: Error finding container 8ff090048f5d3819a9fa39d1941bf7fe3079aad69fe8b34409ace443a8db92c1: Status 404 returned error can't find the container with id 8ff090048f5d3819a9fa39d1941bf7fe3079aad69fe8b34409ace443a8db92c1
Sep 16 13:31:08.973077 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.973043    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:31:08.973173 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.973110    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:31:08.973173 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:08.973139    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:31:09.478198 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:09.478160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" event={"ID":"6ac20dd4-6c19-4382-84b8-bfb05c74ae58","Type":"ContainerStarted","Data":"e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b"}
Sep 16 13:31:09.478703 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:09.478253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" event={"ID":"6ac20dd4-6c19-4382-84b8-bfb05c74ae58","Type":"ContainerStarted","Data":"8ff090048f5d3819a9fa39d1941bf7fe3079aad69fe8b34409ace443a8db92c1"}
Sep 16 13:31:09.503295 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:09.503233    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" podStartSLOduration=1.503217143 podStartE2EDuration="1.503217143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:09.499906287 +0000 UTC m=+1340.922647867" watchObservedRunningTime="2026-09-16 13:31:09.503217143 +0000 UTC m=+1340.925958721"
Sep 16 13:31:09.688139 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:09.688101    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:10.693194 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:10.693156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:11.490544 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:11.490514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:11.491534 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:11.491509    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:16.202712 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.202678    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"]
Sep 16 13:31:16.207540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.207515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.210020 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.209997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-7841d9-serving-cert\""
Sep 16 13:31:16.217778 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.217753    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"]
Sep 16 13:31:16.321345 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.321306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.321543 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.321465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea8893-dba1-4b91-a75b-3f2047180124-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.321543 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.321516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.321659 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.321597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67cp\" (UniqueName: \"kubernetes.io/projected/fcea8893-dba1-4b91-a75b-3f2047180124-kube-api-access-t67cp\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.422363 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.422328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.422551 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.422423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea8893-dba1-4b91-a75b-3f2047180124-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.422551 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.422468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.422551 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.422501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67cp\" (UniqueName: \"kubernetes.io/projected/fcea8893-dba1-4b91-a75b-3f2047180124-kube-api-access-t67cp\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.422794 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:31:16.422620    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-serving-cert: secret "maas-api-e2e-shared-b-w2-7841d9-serving-cert" not found
Sep 16 13:31:16.422794 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:31:16.422726    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls podName:fcea8893-dba1-4b91-a75b-3f2047180124 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:16.922704292 +0000 UTC m=+1348.345445866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls") pod "maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz" (UID: "fcea8893-dba1-4b91-a75b-3f2047180124") : secret "maas-api-e2e-shared-b-w2-7841d9-serving-cert" not found
Sep 16 13:31:16.423102 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.423075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea8893-dba1-4b91-a75b-3f2047180124-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.424867 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.424824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.430427 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.430402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67cp\" (UniqueName: \"kubernetes.io/projected/fcea8893-dba1-4b91-a75b-3f2047180124-kube-api-access-t67cp\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.929333 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.928752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:16.932529 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:16.932473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:17.122169 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:17.122124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:17.400309 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:17.400279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"]
Sep 16 13:31:17.402181 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:31:17.402147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcea8893_dba1_4b91_a75b_3f2047180124.slice/crio-7a7525cb79b168bd4012640ee105c51110431b6abb2244f2e09d1906e4966ae8 WatchSource:0}: Error finding container 7a7525cb79b168bd4012640ee105c51110431b6abb2244f2e09d1906e4966ae8: Status 404 returned error can't find the container with id 7a7525cb79b168bd4012640ee105c51110431b6abb2244f2e09d1906e4966ae8
Sep 16 13:31:17.526418 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:17.526377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz" event={"ID":"fcea8893-dba1-4b91-a75b-3f2047180124","Type":"ContainerStarted","Data":"7a7525cb79b168bd4012640ee105c51110431b6abb2244f2e09d1906e4966ae8"}
Sep 16 13:31:18.534537 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:18.534499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz" event={"ID":"fcea8893-dba1-4b91-a75b-3f2047180124","Type":"ContainerStarted","Data":"ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b"}
Sep 16 13:31:18.535305 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:18.534684    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:18.552872 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:18.552806    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz" podStartSLOduration=2.184986431 podStartE2EDuration="2.552790471s" totalImagesPullingTime="367.80404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:18.550718332 +0000 UTC m=+1349.973459913" watchObservedRunningTime="2026-09-16 13:31:18.552790471 +0000 UTC m=+1349.975532050"
Sep 16 13:31:24.545273 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:24.545245    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:31:38.655247 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:38.655140    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerID="e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b" exitCode=0
Sep 16 13:31:38.655247 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:38.655220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" event={"ID":"6ac20dd4-6c19-4382-84b8-bfb05c74ae58","Type":"ContainerDied","Data":"e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b"}
Sep 16 13:31:38.655832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:38.655812    2577 scope.go:117] "RemoveContainer" containerID="e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b"
Sep 16 13:31:38.657399 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:38.657370    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:31:38.657501 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:38.657445    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:31:38.657501 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:38.657477    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:31:39.662996 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:39.662960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" event={"ID":"6ac20dd4-6c19-4382-84b8-bfb05c74ae58","Type":"ContainerStarted","Data":"a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83"}
Sep 16 13:31:39.687303 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:39.687276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:39.688927 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:39.688895    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 13:31:39.689058 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:39.688968    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 13:31:40.688325 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:40.688291    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 13:31:40.688721 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:40.688344    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 13:31:41.491391 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:41.491352    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:41.687462 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:41.687431    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 13:31:41.687633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:41.687486    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 13:31:42.688081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:42.688035    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 13:31:42.688472 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:42.688104    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 13:31:43.687449 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:43.687411    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 13:31:43.687625 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:43.687475    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 13:31:44.688501 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:44.688462    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 13:31:44.689010 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:44.688521    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 13:31:45.691511 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:45.691474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:31:45.692445 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:31:45.692423    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:32:19.967336 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:19.967298    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"]
Sep 16 13:32:19.967956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:19.967531    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy" containerID="cri-o://c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2" gracePeriod=30
Sep 16 13:32:25.225674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.225648    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:32:25.386475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386095    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-credential-socket\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.386475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-token\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.386475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386222    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istiod-ca-cert\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.386475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386280    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64x4m\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-kube-api-access-64x4m\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.386475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-socket\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.386475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-certs\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.386475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386412    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-podinfo\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.386475 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-envoy\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.387027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-data\") pod \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\" (UID: \"ab5b0ea4-eb39-493b-8c35-1a9e3918726b\") "
Sep 16 13:32:25.387027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386608    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:25.387027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386672    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:25.387027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386703    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:32:25.387027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386871    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:25.387027 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.386918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:25.387287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.387069    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.387287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.387083    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.387287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.387092    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.387287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.387104    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.387287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.387116    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.389164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.389124    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:32:25.389291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.389178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:32:25.389291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.389207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:25.389291 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.389221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-kube-api-access-64x4m" (OuterVolumeSpecName: "kube-api-access-64x4m") pod "ab5b0ea4-eb39-493b-8c35-1a9e3918726b" (UID: "ab5b0ea4-eb39-493b-8c35-1a9e3918726b"). InnerVolumeSpecName "kube-api-access-64x4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:32:25.488297 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.488260    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.488297 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.488296    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64x4m\" (UniqueName: \"kubernetes.io/projected/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-kube-api-access-64x4m\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.488297 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.488307    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.488554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.488316    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5b0ea4-eb39-493b-8c35-1a9e3918726b-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:32:25.930918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.930882    2577 generic.go:356] "Generic (PLEG): container finished" podID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerID="c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2" exitCode=0
Sep 16 13:32:25.931092 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.930930    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"
Sep 16 13:32:25.931092 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.930968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" event={"ID":"ab5b0ea4-eb39-493b-8c35-1a9e3918726b","Type":"ContainerDied","Data":"c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2"}
Sep 16 13:32:25.931092 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.931020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9" event={"ID":"ab5b0ea4-eb39-493b-8c35-1a9e3918726b","Type":"ContainerDied","Data":"2afea5d0cd9b64d6473a628229c85d3b80d668c54797b8020844f9c584a0295d"}
Sep 16 13:32:25.931092 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.931043    2577 scope.go:117] "RemoveContainer" containerID="c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2"
Sep 16 13:32:25.941724 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.941705    2577 scope.go:117] "RemoveContainer" containerID="f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183"
Sep 16 13:32:25.950438 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.950413    2577 scope.go:117] "RemoveContainer" containerID="c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2"
Sep 16 13:32:25.950713 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:32:25.950692    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2\": container with ID starting with c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2 not found: ID does not exist" containerID="c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2"
Sep 16 13:32:25.950786 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.950722    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2"} err="failed to get container status \"c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2\": rpc error: code = NotFound desc = could not find container \"c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2\": container with ID starting with c330e179ecfcf15051a9f9436b6c910163a82d4a698c54428adc179222876ed2 not found: ID does not exist"
Sep 16 13:32:25.950786 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.950741    2577 scope.go:117] "RemoveContainer" containerID="f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183"
Sep 16 13:32:25.951009 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:32:25.950993    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183\": container with ID starting with f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183 not found: ID does not exist" containerID="f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183"
Sep 16 13:32:25.951063 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.951015    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183"} err="failed to get container status \"f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183\": rpc error: code = NotFound desc = could not find container \"f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183\": container with ID starting with f3b7ae9cebe962b9a6bd0ecd3452107b41e6eedd2c714afdcbdcddb28420f183 not found: ID does not exist"
Sep 16 13:32:25.955568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.955545    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"]
Sep 16 13:32:25.962731 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:25.962707    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-48adf5-openshift-default-c6c9b67bb-8csr9"]
Sep 16 13:32:27.110222 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:32:27.110189    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" path="/var/lib/kubelet/pods/ab5b0ea4-eb39-493b-8c35-1a9e3918726b/volumes"
Sep 16 13:36:24.477147 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:24.477108    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"]
Sep 16 13:36:39.496778 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.496721    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j" podUID="65b24411-5fec-4d9c-a930-d659640ae0d1" containerName="main" containerID="cri-o://72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b" gracePeriod=45
Sep 16 13:36:39.755759 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.755699    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:36:39.838228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-dshm\") pod \"65b24411-5fec-4d9c-a930-d659640ae0d1\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") "
Sep 16 13:36:39.838415 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-model-cache\") pod \"65b24411-5fec-4d9c-a930-d659640ae0d1\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") "
Sep 16 13:36:39.838415 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838300    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b24411-5fec-4d9c-a930-d659640ae0d1-tls-certs\") pod \"65b24411-5fec-4d9c-a930-d659640ae0d1\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") "
Sep 16 13:36:39.838415 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-home\") pod \"65b24411-5fec-4d9c-a930-d659640ae0d1\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") "
Sep 16 13:36:39.838598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-tmp-dir\") pod \"65b24411-5fec-4d9c-a930-d659640ae0d1\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") "
Sep 16 13:36:39.838598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838490    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjcnc\" (UniqueName: \"kubernetes.io/projected/65b24411-5fec-4d9c-a930-d659640ae0d1-kube-api-access-qjcnc\") pod \"65b24411-5fec-4d9c-a930-d659640ae0d1\" (UID: \"65b24411-5fec-4d9c-a930-d659640ae0d1\") "
Sep 16 13:36:39.838710 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838577    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "65b24411-5fec-4d9c-a930-d659640ae0d1" (UID: "65b24411-5fec-4d9c-a930-d659640ae0d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:39.838710 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-home" (OuterVolumeSpecName: "home") pod "65b24411-5fec-4d9c-a930-d659640ae0d1" (UID: "65b24411-5fec-4d9c-a930-d659640ae0d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:39.838833 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.838814    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65b24411-5fec-4d9c-a930-d659640ae0d1" (UID: "65b24411-5fec-4d9c-a930-d659640ae0d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:39.839061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.839005    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-home\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:39.839061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.839030    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-tmp-dir\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:39.839061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.839044    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-model-cache\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:39.841147 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.841119    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b24411-5fec-4d9c-a930-d659640ae0d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65b24411-5fec-4d9c-a930-d659640ae0d1" (UID: "65b24411-5fec-4d9c-a930-d659640ae0d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:36:39.841147 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.841129    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-dshm" (OuterVolumeSpecName: "dshm") pod "65b24411-5fec-4d9c-a930-d659640ae0d1" (UID: "65b24411-5fec-4d9c-a930-d659640ae0d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:39.841334 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.841179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b24411-5fec-4d9c-a930-d659640ae0d1-kube-api-access-qjcnc" (OuterVolumeSpecName: "kube-api-access-qjcnc") pod "65b24411-5fec-4d9c-a930-d659640ae0d1" (UID: "65b24411-5fec-4d9c-a930-d659640ae0d1"). InnerVolumeSpecName "kube-api-access-qjcnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:36:39.940188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.940144    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjcnc\" (UniqueName: \"kubernetes.io/projected/65b24411-5fec-4d9c-a930-d659640ae0d1-kube-api-access-qjcnc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:39.940188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.940188    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b24411-5fec-4d9c-a930-d659640ae0d1-dshm\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:39.940188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:39.940199    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b24411-5fec-4d9c-a930-d659640ae0d1-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:40.391704 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.391663    2577 generic.go:356] "Generic (PLEG): container finished" podID="65b24411-5fec-4d9c-a930-d659640ae0d1" containerID="72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b" exitCode=0
Sep 16 13:36:40.391890 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.391710    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"
Sep 16 13:36:40.391890 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.391750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j" event={"ID":"65b24411-5fec-4d9c-a930-d659640ae0d1","Type":"ContainerDied","Data":"72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b"}
Sep 16 13:36:40.391890 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.391795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j" event={"ID":"65b24411-5fec-4d9c-a930-d659640ae0d1","Type":"ContainerDied","Data":"d569a519f8f599bb7090b66d8375ef77037a4a6a14944f7c9f78e451e3d9038d"}
Sep 16 13:36:40.391890 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.391813    2577 scope.go:117] "RemoveContainer" containerID="72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b"
Sep 16 13:36:40.402255 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.402094    2577 scope.go:117] "RemoveContainer" containerID="72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b"
Sep 16 13:36:40.402473 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:36:40.402446    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b\": container with ID starting with 72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b not found: ID does not exist" containerID="72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b"
Sep 16 13:36:40.402585 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.402484    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b"} err="failed to get container status \"72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b\": rpc error: code = NotFound desc = could not find container \"72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b\": container with ID starting with 72fcfeb281266bdc69d75308a6c53ac20d5fe9d7595b11f491eca2b6a057c00b not found: ID does not exist"
Sep 16 13:36:40.418945 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.418922    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"]
Sep 16 13:36:40.425908 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:40.425886    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-w4-f27ea8-kserve-854596d86b-6tr6j"]
Sep 16 13:36:41.110592 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:41.110556    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b24411-5fec-4d9c-a930-d659640ae0d1" path="/var/lib/kubelet/pods/65b24411-5fec-4d9c-a930-d659640ae0d1/volumes"
Sep 16 13:36:51.369177 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:51.369140    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"]
Sep 16 13:36:51.369642 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:51.369468    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy" containerID="cri-o://7958ba326e44a017f13e2d744bb32fdcfe9bbddd8dd24406b9440b6b919e7047" gracePeriod=30
Sep 16 13:36:52.346846 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.346809    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"]
Sep 16 13:36:52.347358 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.347344    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy"
Sep 16 13:36:52.347408 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.347360    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy"
Sep 16 13:36:52.347408 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.347382    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy"
Sep 16 13:36:52.347408 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.347388    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy"
Sep 16 13:36:52.347408 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.347406    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b24411-5fec-4d9c-a930-d659640ae0d1" containerName="main"
Sep 16 13:36:52.347548 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.347412    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b24411-5fec-4d9c-a930-d659640ae0d1" containerName="main"
Sep 16 13:36:52.347548 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.347501    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b24411-5fec-4d9c-a930-d659640ae0d1" containerName="main"
Sep 16 13:36:52.347548 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.347508    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy"
Sep 16 13:36:52.350588 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.350571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.370524 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.370498    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-7e7bb4-openshift-default-dockercfg-xv4jf\""
Sep 16 13:36:52.452084 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.452051    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"]
Sep 16 13:36:52.472145 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjqs\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-kube-api-access-qcjqs\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.472348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-socket\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.472348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-envoy\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.472348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6d8786-cecf-4069-bd3f-75a654b03198-istiod-ca-cert\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.472348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-certs\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.472523 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-podinfo\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.472523 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-credential-socket\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.472523 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-data\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.472660 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.472546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-token\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.573745 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.573711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-credential-socket\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.573927 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.573762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-data\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.573927 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.573812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-token\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.573927 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.573844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjqs\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-kube-api-access-qcjqs\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.573927 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.573906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-socket\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574179 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.573941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-envoy\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574179 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.574019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6d8786-cecf-4069-bd3f-75a654b03198-istiod-ca-cert\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574179 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.574074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-certs\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574179 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.574128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-podinfo\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574398 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.574201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-credential-socket\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574398 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.574295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-data\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574499 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.574402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-socket\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574499 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.574445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-certs\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.574818 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.574801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6d8786-cecf-4069-bd3f-75a654b03198-istiod-ca-cert\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.576436 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.576413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-envoy\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.576665 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.576647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-podinfo\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.583081 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.583058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-token\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.583655 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.583629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjqs\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-kube-api-access-qcjqs\") pod \"e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") " pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.661343 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.661259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:52.948476 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.948447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"]
Sep 16 13:36:52.951690 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:36:52.951663    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6d8786_cecf_4069_bd3f_75a654b03198.slice/crio-601e50251159842719af00d948250d3795aa22117ae253ccecbe3f81466e7a24 WatchSource:0}: Error finding container 601e50251159842719af00d948250d3795aa22117ae253ccecbe3f81466e7a24: Status 404 returned error can't find the container with id 601e50251159842719af00d948250d3795aa22117ae253ccecbe3f81466e7a24
Sep 16 13:36:52.954097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.954060    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:36:52.954202 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.954154    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:36:52.954272 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:52.954211    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:36:53.470471 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:53.470432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" event={"ID":"5c6d8786-cecf-4069-bd3f-75a654b03198","Type":"ContainerStarted","Data":"f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d"}
Sep 16 13:36:53.470948 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:53.470501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" event={"ID":"5c6d8786-cecf-4069-bd3f-75a654b03198","Type":"ContainerStarted","Data":"601e50251159842719af00d948250d3795aa22117ae253ccecbe3f81466e7a24"}
Sep 16 13:36:53.505375 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:53.505320    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podStartSLOduration=1.50530331 podStartE2EDuration="1.50530331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:53.50252864 +0000 UTC m=+1684.925270219" watchObservedRunningTime="2026-09-16 13:36:53.50530331 +0000 UTC m=+1684.928044890"
Sep 16 13:36:53.640643 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:53.640606    2577 patch_prober.go:28] interesting pod/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 13:36:53.640832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:53.640672    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:36:53.662197 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:53.662165    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:54.667681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:54.667647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:55.484098 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:55.484069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:55.485061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:55.485043    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:36:56.491456 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.491422    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerID="7958ba326e44a017f13e2d744bb32fdcfe9bbddd8dd24406b9440b6b919e7047" exitCode=0
Sep 16 13:36:56.491780 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.491488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" event={"ID":"5ac52012-5e9d-4c07-830f-b36b444cb8eb","Type":"ContainerDied","Data":"7958ba326e44a017f13e2d744bb32fdcfe9bbddd8dd24406b9440b6b919e7047"}
Sep 16 13:36:56.491780 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.491549    2577 scope.go:117] "RemoveContainer" containerID="d72d880776a7993c81e6ad72b8007365c56f177165c91c13500f35926e3a575b"
Sep 16 13:36:56.618404 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.618377    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:36:56.720011 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.719979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-podinfo\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-certs\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-socket\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-envoy\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-token\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istiod-ca-cert\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-credential-socket\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720272    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-data\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvlgw\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-kube-api-access-nvlgw\") pod \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\" (UID: \"5ac52012-5e9d-4c07-830f-b36b444cb8eb\") "
Sep 16 13:36:56.720433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:56.720640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720436    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:56.720640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:56.720640 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720584    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:56.720825 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:56.720921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720846    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:56.720921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720889    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:56.720921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720902    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:56.720921 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.720918    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:56.722776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.722750    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:36:56.722776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.722760    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:36:56.722956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.722822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:36:56.722956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.722885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-kube-api-access-nvlgw" (OuterVolumeSpecName: "kube-api-access-nvlgw") pod "5ac52012-5e9d-4c07-830f-b36b444cb8eb" (UID: "5ac52012-5e9d-4c07-830f-b36b444cb8eb"). InnerVolumeSpecName "kube-api-access-nvlgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:36:56.821663 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.821629    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:56.821663 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.821662    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:56.821895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.821673    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:56.821895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.821688    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvlgw\" (UniqueName: \"kubernetes.io/projected/5ac52012-5e9d-4c07-830f-b36b444cb8eb-kube-api-access-nvlgw\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:56.821895 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:56.821704    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac52012-5e9d-4c07-830f-b36b444cb8eb-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:36:57.499211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:57.499180    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"
Sep 16 13:36:57.499674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:57.499229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj" event={"ID":"5ac52012-5e9d-4c07-830f-b36b444cb8eb","Type":"ContainerDied","Data":"1a39b8fbd86247648a0f38d7bd806e7c0987585852a4bdc8dc892fd737bc4c92"}
Sep 16 13:36:57.499674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:57.499296    2577 scope.go:117] "RemoveContainer" containerID="7958ba326e44a017f13e2d744bb32fdcfe9bbddd8dd24406b9440b6b919e7047"
Sep 16 13:36:57.536925 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:57.536898    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"]
Sep 16 13:36:57.554091 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:57.554055    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c71ad7-openshift-default-59c66547b6-tzgxj"]
Sep 16 13:36:59.112429 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:36:59.112395    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" path="/var/lib/kubelet/pods/5ac52012-5e9d-4c07-830f-b36b444cb8eb/volumes"
Sep 16 13:37:16.613554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:16.613474    2577 generic.go:356] "Generic (PLEG): container finished" podID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerID="f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d" exitCode=0
Sep 16 13:37:16.613554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:16.613536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" event={"ID":"5c6d8786-cecf-4069-bd3f-75a654b03198","Type":"ContainerDied","Data":"f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d"}
Sep 16 13:37:16.614017 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:16.613981    2577 scope.go:117] "RemoveContainer" containerID="f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d"
Sep 16 13:37:16.615194 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:16.615164    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:16.615274 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:16.615230    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:16.615274 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:16.615260    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:17.621186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:17.621151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" event={"ID":"5c6d8786-cecf-4069-bd3f-75a654b03198","Type":"ContainerStarted","Data":"8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6"}
Sep 16 13:37:17.662207 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:17.662174    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:37:17.662430 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:17.662398    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 16 13:37:17.662493 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:17.662465    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 16 13:37:18.662064 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:18.662024    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 16 13:37:18.662473 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:18.662084    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 16 13:37:19.662066 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:19.662012    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 16 13:37:19.662708 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:19.662638    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 16 13:37:20.661692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:20.661646    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 16 13:37:20.661888 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:20.661711    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 16 13:37:21.661930 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:21.661892    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 16 13:37:21.662319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:21.661949    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 16 13:37:22.661753 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:22.661712    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 16 13:37:22.661983 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:22.661776    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 16 13:37:23.666125 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:23.666096    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:37:23.666507 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:23.666363    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:37:23.667225 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:23.667202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:37:26.189617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.189586    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"]
Sep 16 13:37:26.190176 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.190158    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy"
Sep 16 13:37:26.190176 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.190176    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy"
Sep 16 13:37:26.190303 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.190292    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy"
Sep 16 13:37:26.190350 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.190308    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5b0ea4-eb39-493b-8c35-1a9e3918726b" containerName="istio-proxy"
Sep 16 13:37:26.192722 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.192703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.195185 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.195162    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8c3cdf-openshift-default-dockercfg-nvtbb\""
Sep 16 13:37:26.208489 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.208464    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"]
Sep 16 13:37:26.241370 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-token\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.241522 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-socket\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.241522 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r66pq\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-kube-api-access-r66pq\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.241522 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-data\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.241633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-certs\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.241633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-podinfo\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.241633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-envoy\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.241729 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istiod-ca-cert\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.241729 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.241653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-credential-socket\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342389 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-token\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-socket\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r66pq\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-kube-api-access-r66pq\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-data\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342598 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-certs\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-podinfo\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-envoy\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istiod-ca-cert\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-credential-socket\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.342842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-socket\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.343141 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.342968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-data\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.343141 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.343029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-certs\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.343238 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.343146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-credential-socket\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.343417 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.343400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istiod-ca-cert\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.344903 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.344885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-envoy\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.345129 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.345111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-podinfo\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.354618 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.354597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r66pq\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-kube-api-access-r66pq\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.355429 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.355403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-token\") pod \"e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") " pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.503918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.503880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:26.785842 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.785728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"]
Sep 16 13:37:26.794629 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.794593    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:26.794782 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.794669    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:26.794782 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:26.794701    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:27.692358 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:27.692315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" event={"ID":"95c6d623-32d0-4dc8-88dd-f16c7a2e872a","Type":"ContainerStarted","Data":"21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f"}
Sep 16 13:37:27.692756 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:27.692370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" event={"ID":"95c6d623-32d0-4dc8-88dd-f16c7a2e872a","Type":"ContainerStarted","Data":"70532e08adc07f427dc22421ea9ceb5d7019b2d76aa01e6897352c084b5a43ff"}
Sep 16 13:37:27.715061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:27.715010    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" podStartSLOduration=1.714994177 podStartE2EDuration="1.714994177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:27.713488896 +0000 UTC m=+1719.136230479" watchObservedRunningTime="2026-09-16 13:37:27.714994177 +0000 UTC m=+1719.137735757"
Sep 16 13:37:28.504343 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:28.504306    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:28.510233 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:28.510206    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:28.698203 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:28.698173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:28.699398 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:28.699365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:50.828678 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:50.828569    2577 generic.go:356] "Generic (PLEG): container finished" podID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerID="21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f" exitCode=0
Sep 16 13:37:50.829109 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:50.828659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" event={"ID":"95c6d623-32d0-4dc8-88dd-f16c7a2e872a","Type":"ContainerDied","Data":"21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f"}
Sep 16 13:37:50.829207 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:50.829193    2577 scope.go:117] "RemoveContainer" containerID="21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f"
Sep 16 13:37:50.830345 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:50.830320    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:50.830432 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:50.830386    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:50.830432 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:50.830417    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:37:51.839501 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:51.839449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" event={"ID":"95c6d623-32d0-4dc8-88dd-f16c7a2e872a","Type":"ContainerStarted","Data":"6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc"}
Sep 16 13:37:52.504677 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:52.504640    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:52.504894 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:52.504826    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 13:37:52.504951 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:52.504914    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 13:37:53.504825 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:53.504791    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 13:37:53.505233 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:53.504842    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 13:37:54.504282 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:54.504242    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 13:37:54.504471 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:54.504311    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 13:37:55.504942 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:55.504905    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 13:37:55.505333 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:55.504961    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 13:37:56.505348 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:56.505309    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 13:37:56.505897 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:56.505477    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 13:37:57.509155 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:57.509127    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:57.509553 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:57.509405    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:37:57.510026 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:37:57.510007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:41:20.150978 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:20.150944    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"]
Sep 16 13:41:20.151884 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:20.151157    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy" containerID="cri-o://a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7" gracePeriod=30
Sep 16 13:41:25.391160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.391135    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:41:25.454996 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.454968    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-token\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455014    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-socket\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455035    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istiod-ca-cert\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455055    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jmh2\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-kube-api-access-8jmh2\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-credential-socket\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455164 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-envoy\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455421 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-podinfo\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455421 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455239    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-data\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455421 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455291    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:41:25.455421 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-certs\") pod \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\" (UID: \"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9\") "
Sep 16 13:41:25.455648 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455462    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:41:25.455648 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:41:25.455648 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455635    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:41:25.455648 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:41:25.455892 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455824    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:25.455892 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455842    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:25.455892 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455871    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:25.455892 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455886    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:25.456113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.455899    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:25.457379 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.457347    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:41:25.457379 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.457357    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:41:25.457537 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.457456    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-kube-api-access-8jmh2" (OuterVolumeSpecName: "kube-api-access-8jmh2") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "kube-api-access-8jmh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:41:25.457537 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.457470    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" (UID: "fbf29a3b-e178-4d37-9a2e-421e8c8d00c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:41:25.557397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.557355    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:25.557397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.557390    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:25.557397 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.557400    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jmh2\" (UniqueName: \"kubernetes.io/projected/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-kube-api-access-8jmh2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:25.557637 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:25.557411    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:41:26.076402 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.076368    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerID="a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7" exitCode=0
Sep 16 13:41:26.076614 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.076419    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"
Sep 16 13:41:26.076614 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.076453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" event={"ID":"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9","Type":"ContainerDied","Data":"a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7"}
Sep 16 13:41:26.076614 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.076504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr" event={"ID":"fbf29a3b-e178-4d37-9a2e-421e8c8d00c9","Type":"ContainerDied","Data":"9635c0b8ee6f395333b5e82c470184ef166229984c31f8e60954c9d4005ba477"}
Sep 16 13:41:26.076614 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.076526    2577 scope.go:117] "RemoveContainer" containerID="a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7"
Sep 16 13:41:26.087318 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.087302    2577 scope.go:117] "RemoveContainer" containerID="1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba"
Sep 16 13:41:26.095494 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.095475    2577 scope.go:117] "RemoveContainer" containerID="a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7"
Sep 16 13:41:26.099085 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:41:26.098509    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7\": container with ID starting with a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7 not found: ID does not exist" containerID="a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7"
Sep 16 13:41:26.099085 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.098552    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7"} err="failed to get container status \"a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7\": rpc error: code = NotFound desc = could not find container \"a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7\": container with ID starting with a27d30a573b5a80fb3769c189fc8ba584bdc103290c31a299b63607653899cf7 not found: ID does not exist"
Sep 16 13:41:26.099085 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.098576    2577 scope.go:117] "RemoveContainer" containerID="1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba"
Sep 16 13:41:26.099365 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:41:26.099343    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba\": container with ID starting with 1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba not found: ID does not exist" containerID="1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba"
Sep 16 13:41:26.099418 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.099375    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba"} err="failed to get container status \"1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba\": rpc error: code = NotFound desc = could not find container \"1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba\": container with ID starting with 1bf86203bffe80a2039a28c417d3aa6cc9524a4418b15a1f6aa1bab00b69a1ba not found: ID does not exist"
Sep 16 13:41:26.102128 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.102108    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"]
Sep 16 13:41:26.108063 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:26.108042    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f27ea8-openshift-default-55779f9bdb-hhrqr"]
Sep 16 13:41:27.110062 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:41:27.110026    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" path="/var/lib/kubelet/pods/fbf29a3b-e178-4d37-9a2e-421e8c8d00c9/volumes"
Sep 16 13:43:22.588154 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588073    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"]
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588563    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy"
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588575    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy"
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588592    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy"
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588598    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy"
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588608    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy"
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588614    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy"
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588677    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy"
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588685    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac52012-5e9d-4c07-830f-b36b444cb8eb" containerName="istio-proxy"
Sep 16 13:43:22.589773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.588693    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf29a3b-e178-4d37-9a2e-421e8c8d00c9" containerName="istio-proxy"
Sep 16 13:43:22.591057 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.591041    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.595149 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.595128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-744b19-openshift-default-dockercfg-qk6pc\""
Sep 16 13:43:22.638418 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.638365    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"]
Sep 16 13:43:22.699087 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-socket\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.699087 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-data\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.699395 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53730326-f35c-43cd-9eb6-9dba08bf6d33-istiod-ca-cert\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.699395 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9lfz\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-kube-api-access-g9lfz\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.699395 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-token\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.699395 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-podinfo\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.699611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-credential-socket\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.699611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-certs\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.699611 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.699555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-envoy\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-credential-socket\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801061 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-certs\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-envoy\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-socket\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-data\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53730326-f35c-43cd-9eb6-9dba08bf6d33-istiod-ca-cert\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9lfz\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-kube-api-access-g9lfz\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801302 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-token\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-podinfo\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-credential-socket\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-certs\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801617 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-data\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801789 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-socket\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.801939 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.801918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53730326-f35c-43cd-9eb6-9dba08bf6d33-istiod-ca-cert\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.803588 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.803567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-envoy\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.803702 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.803650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-podinfo\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.809653 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.809614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9lfz\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-kube-api-access-g9lfz\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.810092 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.810074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-token\") pod \"e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") " pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:22.901434 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:22.901344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:23.189481 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:23.189453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"]
Sep 16 13:43:23.191128 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:43:23.191102    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53730326_f35c_43cd_9eb6_9dba08bf6d33.slice/crio-2d03fc71de3a318dc8bbef26ef0096b0b23d89cff5ea5505b6c16f7be5771b42 WatchSource:0}: Error finding container 2d03fc71de3a318dc8bbef26ef0096b0b23d89cff5ea5505b6c16f7be5771b42: Status 404 returned error can't find the container with id 2d03fc71de3a318dc8bbef26ef0096b0b23d89cff5ea5505b6c16f7be5771b42
Sep 16 13:43:23.193567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:23.193530    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:23.193674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:23.193595    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:23.193674 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:23.193623    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:23.741136 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:23.741100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" event={"ID":"53730326-f35c-43cd-9eb6-9dba08bf6d33","Type":"ContainerStarted","Data":"c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85"}
Sep 16 13:43:23.741511 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:23.741186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" event={"ID":"53730326-f35c-43cd-9eb6-9dba08bf6d33","Type":"ContainerStarted","Data":"2d03fc71de3a318dc8bbef26ef0096b0b23d89cff5ea5505b6c16f7be5771b42"}
Sep 16 13:43:23.770450 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:23.770385    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" podStartSLOduration=1.770371116 podStartE2EDuration="1.770371116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:23.769380416 +0000 UTC m=+2075.192122022" watchObservedRunningTime="2026-09-16 13:43:23.770371116 +0000 UTC m=+2075.193112696"
Sep 16 13:43:23.902154 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:23.902108    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:24.906913 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:24.906879    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:25.752370 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:25.752336    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:25.753384 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:25.753360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:32.652422 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.652392    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"]
Sep 16 13:43:32.655259 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.655242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.659271 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.659248    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-744b19-serving-cert\""
Sep 16 13:43:32.691012 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.690980    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"]
Sep 16 13:43:32.710228 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.710197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-metrics-certs\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.710399 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.710279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqzcl\" (UniqueName: \"kubernetes.io/projected/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-kube-api-access-xqzcl\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.710399 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.710304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.710399 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.710329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.811315 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.811285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-metrics-certs\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.811492 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.811398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzcl\" (UniqueName: \"kubernetes.io/projected/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-kube-api-access-xqzcl\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.811492 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.811428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.811492 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.811463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.811663 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:43:32.811561    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-serving-cert: secret "maas-api-e2e-api-a-744b19-serving-cert" not found
Sep 16 13:43:32.811663 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:43:32.811643    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls podName:d5ee5e3c-4431-4bba-9e80-922c7fc873a0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:33.31162154 +0000 UTC m=+2084.734363099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls") pod "maas-api-e2e-api-a-744b19-5d7fd65664-xghsj" (UID: "d5ee5e3c-4431-4bba-9e80-922c7fc873a0") : secret "maas-api-e2e-api-a-744b19-serving-cert" not found
Sep 16 13:43:32.812162 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.812143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.813820 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.813796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-metrics-certs\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:32.828209 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:32.828184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqzcl\" (UniqueName: \"kubernetes.io/projected/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-kube-api-access-xqzcl\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:33.316330 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:33.316296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:33.318738 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:33.318707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls\") pod \"maas-api-e2e-api-a-744b19-5d7fd65664-xghsj\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:33.566614 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:33.566530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:33.867540 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:33.867462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"]
Sep 16 13:43:33.870431 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:43:33.870400    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ee5e3c_4431_4bba_9e80_922c7fc873a0.slice/crio-021dfd42992303fc45dfbb8efc8cb9c420d6d446e6c4fab8861caab7c21461a0 WatchSource:0}: Error finding container 021dfd42992303fc45dfbb8efc8cb9c420d6d446e6c4fab8861caab7c21461a0: Status 404 returned error can't find the container with id 021dfd42992303fc45dfbb8efc8cb9c420d6d446e6c4fab8861caab7c21461a0
Sep 16 13:43:33.872263 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:33.872242    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:43:34.806662 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:34.806625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj" event={"ID":"d5ee5e3c-4431-4bba-9e80-922c7fc873a0","Type":"ContainerStarted","Data":"3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413"}
Sep 16 13:43:34.806832 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:34.806714    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj" event={"ID":"d5ee5e3c-4431-4bba-9e80-922c7fc873a0","Type":"ContainerStarted","Data":"021dfd42992303fc45dfbb8efc8cb9c420d6d446e6c4fab8861caab7c21461a0"}
Sep 16 13:43:34.806915 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:34.806832    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:34.857415 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:34.857312    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj" podStartSLOduration=2.135084975 podStartE2EDuration="2.857296287s" totalImagesPullingTime="722.211312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:34.85545548 +0000 UTC m=+2086.278197060" watchObservedRunningTime="2026-09-16 13:43:34.857296287 +0000 UTC m=+2086.280037865"
Sep 16 13:43:40.818144 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:40.818116    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:43:48.899201 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:48.899117    2577 generic.go:356] "Generic (PLEG): container finished" podID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerID="c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85" exitCode=0
Sep 16 13:43:48.899201 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:48.899188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" event={"ID":"53730326-f35c-43cd-9eb6-9dba08bf6d33","Type":"ContainerDied","Data":"c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85"}
Sep 16 13:43:48.899621 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:48.899589    2577 scope.go:117] "RemoveContainer" containerID="c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85"
Sep 16 13:43:48.900740 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:48.900712    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:48.900799 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:48.900782    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:48.900839 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:48.900815    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:49.907118 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:49.907081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" event={"ID":"53730326-f35c-43cd-9eb6-9dba08bf6d33","Type":"ContainerStarted","Data":"d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a"}
Sep 16 13:43:50.901933 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:50.901896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:50.902152 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:50.902130    2577 patch_prober.go:28] interesting pod/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 16 13:43:50.902211 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:50.902193    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 16 13:43:51.901789 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:51.901748    2577 patch_prober.go:28] interesting pod/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 16 13:43:51.902240 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:51.901814    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 16 13:43:52.902615 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:52.902575    2577 patch_prober.go:28] interesting pod/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 16 13:43:52.903047 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:52.902631    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 16 13:43:53.902410 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:53.902372    2577 patch_prober.go:28] interesting pod/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 16 13:43:53.902734 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:53.902439    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 16 13:43:54.902142 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:54.902100    2577 patch_prober.go:28] interesting pod/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 16 13:43:54.902352 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:54.902172    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 16 13:43:55.752396 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:55.752361    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:55.906045 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:55.906012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:55.947424 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:55.947399    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:43:57.303188 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.303153    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"]
Sep 16 13:43:57.306130 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.306113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.312672 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.312640    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-804072-openshift-default-dockercfg-h2hz4\""
Sep 16 13:43:57.384593 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.384563    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"]
Sep 16 13:43:57.390058 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-credential-socket\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.390215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-token\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.390215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-socket\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.390215 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-data\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.390384 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-envoy\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.390384 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-podinfo\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.390384 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcjs\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-kube-api-access-hkcjs\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.390384 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istiod-ca-cert\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.390525 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.390467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-certs\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.491470 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.491425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-socket\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.491470 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.491475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-data\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.491732 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.491537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-envoy\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.491732 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.491560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-podinfo\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.491732 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.491585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcjs\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-kube-api-access-hkcjs\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.491732 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.491628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istiod-ca-cert\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.491966 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.491846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-certs\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.491966 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.491959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-credential-socket\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.492075 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.492016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-socket\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.492075 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.492032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-token\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.492259 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.492235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-certs\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.492445 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.492375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-data\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.492539 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.492480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-credential-socket\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.492644 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.492624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istiod-ca-cert\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.494089 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.494059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-envoy\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.494227 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.494210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-podinfo\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.504511 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.504485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcjs\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-kube-api-access-hkcjs\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.505369 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.505338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-token\") pod \"e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") " pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.616570 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.616479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:57.936204 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.936164    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"]
Sep 16 13:43:57.939037 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:43:57.939007    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3ba3e5_2a66_4698_802b_99be91ec3ace.slice/crio-00f3680b2ded29d019b8b59ebaf4e50acfba87f686dc9da7265fba1aa64021db WatchSource:0}: Error finding container 00f3680b2ded29d019b8b59ebaf4e50acfba87f686dc9da7265fba1aa64021db: Status 404 returned error can't find the container with id 00f3680b2ded29d019b8b59ebaf4e50acfba87f686dc9da7265fba1aa64021db
Sep 16 13:43:57.941419 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.941387    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:57.941486 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.941458    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:57.941533 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.941489    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:43:57.959466 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:57.959439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" event={"ID":"9d3ba3e5-2a66-4698-802b-99be91ec3ace","Type":"ContainerStarted","Data":"00f3680b2ded29d019b8b59ebaf4e50acfba87f686dc9da7265fba1aa64021db"}
Sep 16 13:43:58.966339 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:58.966298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" event={"ID":"9d3ba3e5-2a66-4698-802b-99be91ec3ace","Type":"ContainerStarted","Data":"2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1"}
Sep 16 13:43:59.028833 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:59.028777    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" podStartSLOduration=2.02875895 podStartE2EDuration="2.02875895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:59.026591838 +0000 UTC m=+2110.449333416" watchObservedRunningTime="2026-09-16 13:43:59.02875895 +0000 UTC m=+2110.451500530"
Sep 16 13:43:59.617073 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:59.617037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:59.622313 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:59.622286    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:59.972962 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:59.972922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:43:59.973955 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:43:59.973936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:44:26.140638 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:26.140604    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerID="2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1" exitCode=0
Sep 16 13:44:26.141221 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:26.140692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" event={"ID":"9d3ba3e5-2a66-4698-802b-99be91ec3ace","Type":"ContainerDied","Data":"2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1"}
Sep 16 13:44:26.141328 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:26.141310    2577 scope.go:117] "RemoveContainer" containerID="2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1"
Sep 16 13:44:26.142749 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:26.142720    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:44:26.142826 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:26.142787    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:44:26.142826 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:26.142818    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 13:44:27.163058 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:27.163020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" event={"ID":"9d3ba3e5-2a66-4698-802b-99be91ec3ace","Type":"ContainerStarted","Data":"952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01"}
Sep 16 13:44:27.617258 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:27.617217    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:44:27.618532 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:27.618501    2577 patch_prober.go:28] interesting pod/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 16 13:44:27.618670 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:27.618554    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 13:44:28.617874 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:28.617817    2577 patch_prober.go:28] interesting pod/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 16 13:44:28.618251 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:28.617912    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 13:44:29.617114 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:29.617075    2577 patch_prober.go:28] interesting pod/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 16 13:44:29.617303 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:29.617142    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 13:44:29.973870 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:29.973816    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:44:30.617155 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:30.617121    2577 patch_prober.go:28] interesting pod/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 16 13:44:30.617316 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:30.617177    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 13:44:31.617786 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:31.617745    2577 patch_prober.go:28] interesting pod/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 16 13:44:31.618207 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:31.617800    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 13:44:32.621552 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:32.621519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:44:32.622422 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:32.622401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:44:40.368696 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.368649    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"]
Sep 16 13:44:40.369296 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.368946    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj" podUID="d5ee5e3c-4431-4bba-9e80-922c7fc873a0" containerName="maas-api" containerID="cri-o://3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413" gracePeriod=30
Sep 16 13:44:40.635436 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.635290    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:44:40.704402 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.704368    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls\") pod \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") "
Sep 16 13:44:40.704587 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.704464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-metrics-certs\") pod \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") "
Sep 16 13:44:40.704587 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.704548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-openshift-service-ca\") pod \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") "
Sep 16 13:44:40.704714 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.704607    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqzcl\" (UniqueName: \"kubernetes.io/projected/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-kube-api-access-xqzcl\") pod \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\" (UID: \"d5ee5e3c-4431-4bba-9e80-922c7fc873a0\") "
Sep 16 13:44:40.704938 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.704911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5ee5e3c-4431-4bba-9e80-922c7fc873a0" (UID: "d5ee5e3c-4431-4bba-9e80-922c7fc873a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:44:40.705063 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.705045    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-openshift-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:40.706588 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.706566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5ee5e3c-4431-4bba-9e80-922c7fc873a0" (UID: "d5ee5e3c-4431-4bba-9e80-922c7fc873a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:44:40.706688 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.706656    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5ee5e3c-4431-4bba-9e80-922c7fc873a0" (UID: "d5ee5e3c-4431-4bba-9e80-922c7fc873a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:44:40.706739 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.706722    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-kube-api-access-xqzcl" (OuterVolumeSpecName: "kube-api-access-xqzcl") pod "d5ee5e3c-4431-4bba-9e80-922c7fc873a0" (UID: "d5ee5e3c-4431-4bba-9e80-922c7fc873a0"). InnerVolumeSpecName "kube-api-access-xqzcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:44:40.808840 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.808794    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqzcl\" (UniqueName: \"kubernetes.io/projected/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-kube-api-access-xqzcl\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:40.808840 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.808839    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-maas-api-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:40.809132 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:40.808883    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ee5e3c-4431-4bba-9e80-922c7fc873a0-metrics-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:41.247554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.247514    2577 generic.go:356] "Generic (PLEG): container finished" podID="d5ee5e3c-4431-4bba-9e80-922c7fc873a0" containerID="3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413" exitCode=0
Sep 16 13:44:41.247737 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.247555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj" event={"ID":"d5ee5e3c-4431-4bba-9e80-922c7fc873a0","Type":"ContainerDied","Data":"3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413"}
Sep 16 13:44:41.247737 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.247625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj" event={"ID":"d5ee5e3c-4431-4bba-9e80-922c7fc873a0","Type":"ContainerDied","Data":"021dfd42992303fc45dfbb8efc8cb9c420d6d446e6c4fab8861caab7c21461a0"}
Sep 16 13:44:41.247737 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.247567    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"
Sep 16 13:44:41.247737 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.247660    2577 scope.go:117] "RemoveContainer" containerID="3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413"
Sep 16 13:44:41.257889 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.257875    2577 scope.go:117] "RemoveContainer" containerID="3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413"
Sep 16 13:44:41.258196 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:44:41.258166    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413\": container with ID starting with 3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413 not found: ID does not exist" containerID="3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413"
Sep 16 13:44:41.258311 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.258202    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413"} err="failed to get container status \"3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413\": rpc error: code = NotFound desc = could not find container \"3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413\": container with ID starting with 3c8b0f1767ee973757efe5abfaffa0df7628aece3a23f554456aaf0f45ad4413 not found: ID does not exist"
Sep 16 13:44:41.268456 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.268429    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"]
Sep 16 13:44:41.278658 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:41.278631    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-744b19-5d7fd65664-xghsj"]
Sep 16 13:44:43.110486 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:43.110453    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5ee5e3c-4431-4bba-9e80-922c7fc873a0" path="/var/lib/kubelet/pods/d5ee5e3c-4431-4bba-9e80-922c7fc873a0/volumes"
Sep 16 13:44:47.967221 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:47.967184    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"]
Sep 16 13:44:47.967591 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:47.967455    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy" containerID="cri-o://d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a" gracePeriod=30
Sep 16 13:44:53.235182 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.235122    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:44:53.247827 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.247804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-certs\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.247956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.247878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9lfz\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-kube-api-access-g9lfz\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.247956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.247907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-data\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.247956 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.247943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-credential-socket\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.248111 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.247971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-envoy\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.248111 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-token\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.248111 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-podinfo\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.248111 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248087    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:53.248306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-socket\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.248306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53730326-f35c-43cd-9eb6-9dba08bf6d33-istiod-ca-cert\") pod \"53730326-f35c-43cd-9eb6-9dba08bf6d33\" (UID: \"53730326-f35c-43cd-9eb6-9dba08bf6d33\") "
Sep 16 13:44:53.248306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-data" (OuterVolumeSpecName: "istio-data") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:53.248467 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:53.248909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248588    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:53.248909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53730326-f35c-43cd-9eb6-9dba08bf6d33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:44:53.248909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248799    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53730326-f35c-43cd-9eb6-9dba08bf6d33-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.248909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248819    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.248909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248832    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.248909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248845    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.248909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.248886    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.250333 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.250300    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:53.250487 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.250468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:44:53.250554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.250542    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-token" (OuterVolumeSpecName: "istio-token") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:44:53.250626 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.250609    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-kube-api-access-g9lfz" (OuterVolumeSpecName: "kube-api-access-g9lfz") pod "53730326-f35c-43cd-9eb6-9dba08bf6d33" (UID: "53730326-f35c-43cd-9eb6-9dba08bf6d33"). InnerVolumeSpecName "kube-api-access-g9lfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:44:53.322567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.322536    2577 generic.go:356] "Generic (PLEG): container finished" podID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerID="d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a" exitCode=0
Sep 16 13:44:53.322909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.322585    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"
Sep 16 13:44:53.322909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.322621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" event={"ID":"53730326-f35c-43cd-9eb6-9dba08bf6d33","Type":"ContainerDied","Data":"d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a"}
Sep 16 13:44:53.322909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.322676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k" event={"ID":"53730326-f35c-43cd-9eb6-9dba08bf6d33","Type":"ContainerDied","Data":"2d03fc71de3a318dc8bbef26ef0096b0b23d89cff5ea5505b6c16f7be5771b42"}
Sep 16 13:44:53.322909 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.322705    2577 scope.go:117] "RemoveContainer" containerID="d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a"
Sep 16 13:44:53.332969 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.332948    2577 scope.go:117] "RemoveContainer" containerID="c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85"
Sep 16 13:44:53.341925 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.341902    2577 scope.go:117] "RemoveContainer" containerID="d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a"
Sep 16 13:44:53.342220 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:44:53.342201    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a\": container with ID starting with d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a not found: ID does not exist" containerID="d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a"
Sep 16 13:44:53.342287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.342227    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a"} err="failed to get container status \"d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a\": rpc error: code = NotFound desc = could not find container \"d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a\": container with ID starting with d7be07678c83ff7edc02b623d8ff34d748822956d9037bd6a29e832537b9c68a not found: ID does not exist"
Sep 16 13:44:53.342287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.342244    2577 scope.go:117] "RemoveContainer" containerID="c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85"
Sep 16 13:44:53.342455 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:44:53.342439    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85\": container with ID starting with c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85 not found: ID does not exist" containerID="c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85"
Sep 16 13:44:53.342497 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.342458    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85"} err="failed to get container status \"c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85\": rpc error: code = NotFound desc = could not find container \"c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85\": container with ID starting with c5de71f3b5d17c724cf75047711ad37a9d3e8de681566b3ec9ec07a0a76f2c85 not found: ID does not exist"
Sep 16 13:44:53.349229 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.349207    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"]
Sep 16 13:44:53.349981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.349960    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9lfz\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-kube-api-access-g9lfz\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.350036 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.349988    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.350036 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.349998    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.350036 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.350006    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53730326-f35c-43cd-9eb6-9dba08bf6d33-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:44:53.351667 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:53.351650    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-744b19-openshift-default-6f7c94df4-f8h2k"]
Sep 16 13:44:55.111218 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:44:55.111175    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" path="/var/lib/kubelet/pods/53730326-f35c-43cd-9eb6-9dba08bf6d33/volumes"
Sep 16 13:45:11.720692 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:11.720654    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"]
Sep 16 13:45:11.721310 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:11.720945    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy" containerID="cri-o://952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01" gracePeriod=30
Sep 16 13:45:16.995495 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:16.995430    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:45:17.108377 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-envoy\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.108571 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istiod-ca-cert\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.108571 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108426    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkcjs\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-kube-api-access-hkcjs\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.108571 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-data\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.108571 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-credential-socket\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.108783 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108615    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-podinfo\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.108783 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108645    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-token\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.108783 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108667    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-certs\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.108783 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108713    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-socket\") pod \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\" (UID: \"9d3ba3e5-2a66-4698-802b-99be91ec3ace\") "
Sep 16 13:45:17.109003 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108892    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:17.109003 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.108928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:17.109161 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.109039    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:17.109161 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.109112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:17.109161 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.109128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:17.109277 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.109186    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.109277 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.109202    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.111294 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.111170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:45:17.111294 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.111248    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:17.111294 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.111259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-kube-api-access-hkcjs" (OuterVolumeSpecName: "kube-api-access-hkcjs") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "kube-api-access-hkcjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:17.111538 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.111441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d3ba3e5-2a66-4698-802b-99be91ec3ace" (UID: "9d3ba3e5-2a66-4698-802b-99be91ec3ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:17.210760 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.210728    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.210760 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.210757    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.210760 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.210768    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.211011 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.210776    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.211011 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.210785    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.211011 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.210793    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ba3e5-2a66-4698-802b-99be91ec3ace-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.211011 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.210801    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcjs\" (UniqueName: \"kubernetes.io/projected/9d3ba3e5-2a66-4698-802b-99be91ec3ace-kube-api-access-hkcjs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:17.469789 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.469753    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerID="952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01" exitCode=0
Sep 16 13:45:17.469981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.469806    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"
Sep 16 13:45:17.469981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.469824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" event={"ID":"9d3ba3e5-2a66-4698-802b-99be91ec3ace","Type":"ContainerDied","Data":"952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01"}
Sep 16 13:45:17.469981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.469896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb" event={"ID":"9d3ba3e5-2a66-4698-802b-99be91ec3ace","Type":"ContainerDied","Data":"00f3680b2ded29d019b8b59ebaf4e50acfba87f686dc9da7265fba1aa64021db"}
Sep 16 13:45:17.469981 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.469926    2577 scope.go:117] "RemoveContainer" containerID="952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01"
Sep 16 13:45:17.479568 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.479552    2577 scope.go:117] "RemoveContainer" containerID="2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1"
Sep 16 13:45:17.488457 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.488432    2577 scope.go:117] "RemoveContainer" containerID="952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01"
Sep 16 13:45:17.488787 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:45:17.488767    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01\": container with ID starting with 952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01 not found: ID does not exist" containerID="952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01"
Sep 16 13:45:17.488887 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.488796    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01"} err="failed to get container status \"952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01\": rpc error: code = NotFound desc = could not find container \"952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01\": container with ID starting with 952299e8260e56cd9ccaf83bb526cda7371836f9c80a8416bf4d2741cac55c01 not found: ID does not exist"
Sep 16 13:45:17.488887 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.488815    2577 scope.go:117] "RemoveContainer" containerID="2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1"
Sep 16 13:45:17.489130 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:45:17.489112    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1\": container with ID starting with 2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1 not found: ID does not exist" containerID="2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1"
Sep 16 13:45:17.489186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.489129    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"]
Sep 16 13:45:17.489186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.489138    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1"} err="failed to get container status \"2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1\": rpc error: code = NotFound desc = could not find container \"2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1\": container with ID starting with 2477e3c80ac14c41d30f783f4b89b98fbaa743bea7e52c680f5d51a45f066bf1 not found: ID does not exist"
Sep 16 13:45:17.494183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:17.494162    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-804072-openshift-default-6797dc6bf6-bnsbb"]
Sep 16 13:45:19.119232 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:19.119196    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" path="/var/lib/kubelet/pods/9d3ba3e5-2a66-4698-802b-99be91ec3ace/volumes"
Sep 16 13:45:22.685652 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.685612    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"]
Sep 16 13:45:22.686145 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.685924    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j" podUID="9445f919-a104-41fc-ae2e-cdc67a0de187" containerName="maas-api" containerID="cri-o://2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23" gracePeriod=30
Sep 16 13:45:22.956491 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.956462    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:45:22.972072 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.972035    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls\") pod \"9445f919-a104-41fc-ae2e-cdc67a0de187\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") "
Sep 16 13:45:22.972072 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.972075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-metrics-certs\") pod \"9445f919-a104-41fc-ae2e-cdc67a0de187\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") "
Sep 16 13:45:22.972287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.972107    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445f919-a104-41fc-ae2e-cdc67a0de187-openshift-service-ca\") pod \"9445f919-a104-41fc-ae2e-cdc67a0de187\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") "
Sep 16 13:45:22.972287 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.972141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn4hs\" (UniqueName: \"kubernetes.io/projected/9445f919-a104-41fc-ae2e-cdc67a0de187-kube-api-access-hn4hs\") pod \"9445f919-a104-41fc-ae2e-cdc67a0de187\" (UID: \"9445f919-a104-41fc-ae2e-cdc67a0de187\") "
Sep 16 13:45:22.972563 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.972531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9445f919-a104-41fc-ae2e-cdc67a0de187-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9445f919-a104-41fc-ae2e-cdc67a0de187" (UID: "9445f919-a104-41fc-ae2e-cdc67a0de187"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:22.974289 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.974263    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9445f919-a104-41fc-ae2e-cdc67a0de187" (UID: "9445f919-a104-41fc-ae2e-cdc67a0de187"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:22.974418 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.974306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9445f919-a104-41fc-ae2e-cdc67a0de187" (UID: "9445f919-a104-41fc-ae2e-cdc67a0de187"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:22.974418 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:22.974329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9445f919-a104-41fc-ae2e-cdc67a0de187-kube-api-access-hn4hs" (OuterVolumeSpecName: "kube-api-access-hn4hs") pod "9445f919-a104-41fc-ae2e-cdc67a0de187" (UID: "9445f919-a104-41fc-ae2e-cdc67a0de187"). InnerVolumeSpecName "kube-api-access-hn4hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:23.073754 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.073716    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-maas-api-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:23.073754 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.073748    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9445f919-a104-41fc-ae2e-cdc67a0de187-metrics-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:23.073754 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.073757    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445f919-a104-41fc-ae2e-cdc67a0de187-openshift-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:23.074019 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.073766    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4hs\" (UniqueName: \"kubernetes.io/projected/9445f919-a104-41fc-ae2e-cdc67a0de187-kube-api-access-hn4hs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:23.506320 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.506282    2577 generic.go:356] "Generic (PLEG): container finished" podID="9445f919-a104-41fc-ae2e-cdc67a0de187" containerID="2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23" exitCode=0
Sep 16 13:45:23.506538 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.506329    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"
Sep 16 13:45:23.506538 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.506351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j" event={"ID":"9445f919-a104-41fc-ae2e-cdc67a0de187","Type":"ContainerDied","Data":"2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23"}
Sep 16 13:45:23.506538 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.506426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j" event={"ID":"9445f919-a104-41fc-ae2e-cdc67a0de187","Type":"ContainerDied","Data":"94703875fe1dadfb428ca7ce386ab4152923690f609fd62d78d51bef6f507fe3"}
Sep 16 13:45:23.506538 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.506458    2577 scope.go:117] "RemoveContainer" containerID="2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23"
Sep 16 13:45:23.516259 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.516213    2577 scope.go:117] "RemoveContainer" containerID="2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23"
Sep 16 13:45:23.516715 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:45:23.516674    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23\": container with ID starting with 2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23 not found: ID does not exist" containerID="2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23"
Sep 16 13:45:23.516821 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.516727    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23"} err="failed to get container status \"2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23\": rpc error: code = NotFound desc = could not find container \"2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23\": container with ID starting with 2baa8dfc966461e5becb107ebf42a6e5f13623516c74e70b6626a240008e9b23 not found: ID does not exist"
Sep 16 13:45:23.528637 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.528605    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"]
Sep 16 13:45:23.532518 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:23.532490    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0514e-d5d68456f-6sg5j"]
Sep 16 13:45:25.110459 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:25.110421    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9445f919-a104-41fc-ae2e-cdc67a0de187" path="/var/lib/kubelet/pods/9445f919-a104-41fc-ae2e-cdc67a0de187/volumes"
Sep 16 13:45:41.225681 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.225502    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"]
Sep 16 13:45:41.226149 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.225786    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz" podUID="fcea8893-dba1-4b91-a75b-3f2047180124" containerName="maas-api" containerID="cri-o://ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b" gracePeriod=30
Sep 16 13:45:41.478373 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.478309    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:45:41.575221 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.575192    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls\") pod \"fcea8893-dba1-4b91-a75b-3f2047180124\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") "
Sep 16 13:45:41.575399 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.575263    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-metrics-certs\") pod \"fcea8893-dba1-4b91-a75b-3f2047180124\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") "
Sep 16 13:45:41.575399 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.575331    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t67cp\" (UniqueName: \"kubernetes.io/projected/fcea8893-dba1-4b91-a75b-3f2047180124-kube-api-access-t67cp\") pod \"fcea8893-dba1-4b91-a75b-3f2047180124\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") "
Sep 16 13:45:41.575399 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.575378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea8893-dba1-4b91-a75b-3f2047180124-openshift-service-ca\") pod \"fcea8893-dba1-4b91-a75b-3f2047180124\" (UID: \"fcea8893-dba1-4b91-a75b-3f2047180124\") "
Sep 16 13:45:41.575791 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.575761    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcea8893-dba1-4b91-a75b-3f2047180124-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fcea8893-dba1-4b91-a75b-3f2047180124" (UID: "fcea8893-dba1-4b91-a75b-3f2047180124"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:41.577533 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.577491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcea8893-dba1-4b91-a75b-3f2047180124-kube-api-access-t67cp" (OuterVolumeSpecName: "kube-api-access-t67cp") pod "fcea8893-dba1-4b91-a75b-3f2047180124" (UID: "fcea8893-dba1-4b91-a75b-3f2047180124"). InnerVolumeSpecName "kube-api-access-t67cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:41.577533 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.577505    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fcea8893-dba1-4b91-a75b-3f2047180124" (UID: "fcea8893-dba1-4b91-a75b-3f2047180124"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:41.577721 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.577565    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fcea8893-dba1-4b91-a75b-3f2047180124" (UID: "fcea8893-dba1-4b91-a75b-3f2047180124"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:41.632469 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.631777    2577 generic.go:356] "Generic (PLEG): container finished" podID="fcea8893-dba1-4b91-a75b-3f2047180124" containerID="ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b" exitCode=0
Sep 16 13:45:41.632469 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.631886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz" event={"ID":"fcea8893-dba1-4b91-a75b-3f2047180124","Type":"ContainerDied","Data":"ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b"}
Sep 16 13:45:41.632469 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.631935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz" event={"ID":"fcea8893-dba1-4b91-a75b-3f2047180124","Type":"ContainerDied","Data":"7a7525cb79b168bd4012640ee105c51110431b6abb2244f2e09d1906e4966ae8"}
Sep 16 13:45:41.632469 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.631964    2577 scope.go:117] "RemoveContainer" containerID="ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b"
Sep 16 13:45:41.632469 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.631966    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"
Sep 16 13:45:41.654912 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.654883    2577 scope.go:117] "RemoveContainer" containerID="ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b"
Sep 16 13:45:41.655297 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:45:41.655266    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b\": container with ID starting with ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b not found: ID does not exist" containerID="ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b"
Sep 16 13:45:41.655398 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.655309    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b"} err="failed to get container status \"ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b\": rpc error: code = NotFound desc = could not find container \"ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b\": container with ID starting with ba248a1c6a9fd97f5909f38b0f17c22c2986981a9f34d8b589735222e1fcb07b not found: ID does not exist"
Sep 16 13:45:41.662554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.662528    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"]
Sep 16 13:45:41.665372 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.665350    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7841d9-5c84966b46-jjvzz"]
Sep 16 13:45:41.676367 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.676344    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-maas-api-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:41.676367 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.676367    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea8893-dba1-4b91-a75b-3f2047180124-metrics-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:41.676477 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.676377    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t67cp\" (UniqueName: \"kubernetes.io/projected/fcea8893-dba1-4b91-a75b-3f2047180124-kube-api-access-t67cp\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:41.676477 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:41.676385    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea8893-dba1-4b91-a75b-3f2047180124-openshift-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:43.111110 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:43.111068    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcea8893-dba1-4b91-a75b-3f2047180124" path="/var/lib/kubelet/pods/fcea8893-dba1-4b91-a75b-3f2047180124/volumes"
Sep 16 13:45:49.121060 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:49.121023    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"]
Sep 16 13:45:49.121489 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:49.121266    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy" containerID="cri-o://a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83" gracePeriod=30
Sep 16 13:45:54.385362 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.385301    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:45:54.408306 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408277    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-socket\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408485 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408319    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-token\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408485 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-podinfo\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408485 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408372    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-credential-socket\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408485 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v55fd\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-kube-api-access-v55fd\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408485 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408473    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istiod-ca-cert\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408779 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408528    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-data\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408779 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-certs\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408779 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-envoy\") pod \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\" (UID: \"6ac20dd4-6c19-4382-84b8-bfb05c74ae58\") "
Sep 16 13:45:54.408779 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:54.409022 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.408964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:54.409992 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.409102    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.409992 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.409122    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.410384 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.410357    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:54.410472 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.410450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:54.410749 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.410724    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:54.411046 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.411011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:45:54.411165 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.411138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:54.411275 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.411257    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:54.411479 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.411465    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-kube-api-access-v55fd" (OuterVolumeSpecName: "kube-api-access-v55fd") pod "6ac20dd4-6c19-4382-84b8-bfb05c74ae58" (UID: "6ac20dd4-6c19-4382-84b8-bfb05c74ae58"). InnerVolumeSpecName "kube-api-access-v55fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:54.509937 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.509904    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.509937 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.509929    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.509937 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.509938    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.510169 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.509950    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55fd\" (UniqueName: \"kubernetes.io/projected/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-kube-api-access-v55fd\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.510169 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.509960    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.510169 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.509969    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.510169 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.509977    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac20dd4-6c19-4382-84b8-bfb05c74ae58-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:45:54.704944 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.704909    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerID="a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83" exitCode=0
Sep 16 13:45:54.705113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.704953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"
Sep 16 13:45:54.705113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.705000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" event={"ID":"6ac20dd4-6c19-4382-84b8-bfb05c74ae58","Type":"ContainerDied","Data":"a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83"}
Sep 16 13:45:54.705113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.705076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr" event={"ID":"6ac20dd4-6c19-4382-84b8-bfb05c74ae58","Type":"ContainerDied","Data":"8ff090048f5d3819a9fa39d1941bf7fe3079aad69fe8b34409ace443a8db92c1"}
Sep 16 13:45:54.705113 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.705102    2577 scope.go:117] "RemoveContainer" containerID="a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83"
Sep 16 13:45:54.714595 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.714574    2577 scope.go:117] "RemoveContainer" containerID="e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b"
Sep 16 13:45:54.722813 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.722798    2577 scope.go:117] "RemoveContainer" containerID="a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83"
Sep 16 13:45:54.723107 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:45:54.723085    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83\": container with ID starting with a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83 not found: ID does not exist" containerID="a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83"
Sep 16 13:45:54.723170 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.723114    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83"} err="failed to get container status \"a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83\": rpc error: code = NotFound desc = could not find container \"a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83\": container with ID starting with a9d30cf4806118c0abbff4d082b29c82075baa640425093457e54cafe8d94e83 not found: ID does not exist"
Sep 16 13:45:54.723170 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.723131    2577 scope.go:117] "RemoveContainer" containerID="e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b"
Sep 16 13:45:54.723384 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:45:54.723363    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b\": container with ID starting with e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b not found: ID does not exist" containerID="e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b"
Sep 16 13:45:54.723440 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.723394    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b"} err="failed to get container status \"e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b\": rpc error: code = NotFound desc = could not find container \"e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b\": container with ID starting with e5237d6c3906e67e463d28f7e99ab442a0fe0bd007dce6021d4ff6ada552481b not found: ID does not exist"
Sep 16 13:45:54.728377 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.728357    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"]
Sep 16 13:45:54.732013 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:54.731992    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7841d9-openshift-default-99cf7ddbb-bf5vr"]
Sep 16 13:45:55.111918 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:45:55.111831    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" path="/var/lib/kubelet/pods/6ac20dd4-6c19-4382-84b8-bfb05c74ae58/volumes"
Sep 16 13:54:35.765490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:35.765456    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"]
Sep 16 13:54:35.766079 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:35.765665    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy" containerID="cri-o://8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6" gracePeriod=30
Sep 16 13:54:38.668934 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:38.668889    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 13:54:38.669313 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:38.668952    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:54:41.026456 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.026428    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:54:41.141186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-podinfo\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcjqs\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-kube-api-access-qcjqs\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141186 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141185    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-certs\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141480 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-data\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141480 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141271    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-socket\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141480 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-credential-socket\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141480 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-token\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141480 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141361    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6d8786-cecf-4069-bd3f-75a654b03198-istiod-ca-cert\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141480 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-envoy\") pod \"5c6d8786-cecf-4069-bd3f-75a654b03198\" (UID: \"5c6d8786-cecf-4069-bd3f-75a654b03198\") "
Sep 16 13:54:41.141802 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:41.141802 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:41.141802 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:41.141998 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.141912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:41.142354 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.142290    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.142472 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.142407    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.142472 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.142424    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.142472 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.142437    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.142472 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.142307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6d8786-cecf-4069-bd3f-75a654b03198-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:54:41.143548 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.143518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:54:41.143678 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.143655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:54:41.143754 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.143721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-kube-api-access-qcjqs" (OuterVolumeSpecName: "kube-api-access-qcjqs") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "kube-api-access-qcjqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:54:41.143799 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.143776    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c6d8786-cecf-4069-bd3f-75a654b03198" (UID: "5c6d8786-cecf-4069-bd3f-75a654b03198"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:41.243183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.243130    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.243183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.243175    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6d8786-cecf-4069-bd3f-75a654b03198-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.243183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.243185    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.243183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.243192    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6d8786-cecf-4069-bd3f-75a654b03198-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.243183 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.243201    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjqs\" (UniqueName: \"kubernetes.io/projected/5c6d8786-cecf-4069-bd3f-75a654b03198-kube-api-access-qcjqs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:41.569718 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.569686    2577 generic.go:356] "Generic (PLEG): container finished" podID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerID="8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6" exitCode=0
Sep 16 13:54:41.569916 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.569741    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"
Sep 16 13:54:41.569916 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.569769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" event={"ID":"5c6d8786-cecf-4069-bd3f-75a654b03198","Type":"ContainerDied","Data":"8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6"}
Sep 16 13:54:41.569916 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.569823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx" event={"ID":"5c6d8786-cecf-4069-bd3f-75a654b03198","Type":"ContainerDied","Data":"601e50251159842719af00d948250d3795aa22117ae253ccecbe3f81466e7a24"}
Sep 16 13:54:41.569916 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.569846    2577 scope.go:117] "RemoveContainer" containerID="8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6"
Sep 16 13:54:41.580671 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.580654    2577 scope.go:117] "RemoveContainer" containerID="f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d"
Sep 16 13:54:41.593675 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.593654    2577 scope.go:117] "RemoveContainer" containerID="8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6"
Sep 16 13:54:41.594003 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:54:41.593978    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6\": container with ID starting with 8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6 not found: ID does not exist" containerID="8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6"
Sep 16 13:54:41.594063 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.594013    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6"} err="failed to get container status \"8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6\": rpc error: code = NotFound desc = could not find container \"8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6\": container with ID starting with 8cf066bb7c3ff8e945998d541d61e8269c5680d2586324c3079a9e06cac4d8c6 not found: ID does not exist"
Sep 16 13:54:41.594063 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.594032    2577 scope.go:117] "RemoveContainer" containerID="f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d"
Sep 16 13:54:41.594359 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:54:41.594322    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d\": container with ID starting with f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d not found: ID does not exist" containerID="f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d"
Sep 16 13:54:41.594478 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.594363    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d"} err="failed to get container status \"f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d\": rpc error: code = NotFound desc = could not find container \"f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d\": container with ID starting with f06dafe7d2eb2abf48c271fceb4587c7cf345e1dc218503b841efe93f50c9b2d not found: ID does not exist"
Sep 16 13:54:41.595507 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.595449    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"]
Sep 16 13:54:41.597735 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:41.597714    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7e7bb4-openshift-default-7768964f58-l5fqx"]
Sep 16 13:54:43.110133 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:43.110099    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" path="/var/lib/kubelet/pods/5c6d8786-cecf-4069-bd3f-75a654b03198/volumes"
Sep 16 13:54:53.400130 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:53.400093    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"]
Sep 16 13:54:53.400555 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:53.400372    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy" containerID="cri-o://6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc" gracePeriod=30
Sep 16 13:54:57.511550 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:57.511513    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 13:54:57.511954 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:57.511614    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:54:58.659771 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.659740    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:54:58.668790 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.668719    2577 generic.go:356] "Generic (PLEG): container finished" podID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerID="6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc" exitCode=0
Sep 16 13:54:58.668790 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.668762    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"
Sep 16 13:54:58.668953 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.668800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" event={"ID":"95c6d623-32d0-4dc8-88dd-f16c7a2e872a","Type":"ContainerDied","Data":"6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc"}
Sep 16 13:54:58.668953 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.668862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr" event={"ID":"95c6d623-32d0-4dc8-88dd-f16c7a2e872a","Type":"ContainerDied","Data":"70532e08adc07f427dc22421ea9ceb5d7019b2d76aa01e6897352c084b5a43ff"}
Sep 16 13:54:58.668953 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.668884    2577 scope.go:117] "RemoveContainer" containerID="6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc"
Sep 16 13:54:58.677669 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.677650    2577 scope.go:117] "RemoveContainer" containerID="21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f"
Sep 16 13:54:58.687641 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.687616    2577 scope.go:117] "RemoveContainer" containerID="6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc"
Sep 16 13:54:58.687998 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:54:58.687975    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc\": container with ID starting with 6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc not found: ID does not exist" containerID="6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc"
Sep 16 13:54:58.688106 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.688006    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc"} err="failed to get container status \"6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc\": rpc error: code = NotFound desc = could not find container \"6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc\": container with ID starting with 6acc038838d6031ab583dd81aa549ff4d31b864a49411f84e4b4c6874cc817fc not found: ID does not exist"
Sep 16 13:54:58.688106 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.688025    2577 scope.go:117] "RemoveContainer" containerID="21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f"
Sep 16 13:54:58.688325 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:54:58.688307    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f\": container with ID starting with 21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f not found: ID does not exist" containerID="21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f"
Sep 16 13:54:58.688398 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.688329    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f"} err="failed to get container status \"21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f\": rpc error: code = NotFound desc = could not find container \"21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f\": container with ID starting with 21a9661c5ff884efd7350fb5562d6aea5474a4c9e595fb2e9ead57199aa7bc3f not found: ID does not exist"
Sep 16 13:54:58.837969 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.837931    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-socket\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-token\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838040    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-credential-socket\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838077    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-podinfo\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-data\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838160 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-certs\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:58.838433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838293    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-envoy\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:58.838433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-data" (OuterVolumeSpecName: "istio-data") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:58.838433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838428    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r66pq\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-kube-api-access-r66pq\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838679 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istiod-ca-cert\") pod \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\" (UID: \"95c6d623-32d0-4dc8-88dd-f16c7a2e872a\") "
Sep 16 13:54:58.838679 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838472    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:58.838944 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838910    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:54:58.839052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838969    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-credential-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.839052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.838992    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.839052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.839006    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.839052 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.839020    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-workload-socket\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.840278 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.840249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-token" (OuterVolumeSpecName: "istio-token") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:54:58.840456 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.840439    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:58.840717 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.840699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:54:58.840776 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.840719    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-kube-api-access-r66pq" (OuterVolumeSpecName: "kube-api-access-r66pq") pod "95c6d623-32d0-4dc8-88dd-f16c7a2e872a" (UID: "95c6d623-32d0-4dc8-88dd-f16c7a2e872a"). InnerVolumeSpecName "kube-api-access-r66pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:54:58.939931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.939813    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r66pq\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-kube-api-access-r66pq\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.939931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.939847    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istiod-ca-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.939931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.939887    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-token\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.939931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.939896    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-podinfo\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.939931 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.939904    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c6d623-32d0-4dc8-88dd-f16c7a2e872a-istio-envoy\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:54:58.991362 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.991327    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"]
Sep 16 13:54:58.995151 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:58.995123    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c3cdf-openshift-default-59f5b5d584-d9wkr"]
Sep 16 13:54:59.110083 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:54:59.110049    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" path="/var/lib/kubelet/pods/95c6d623-32d0-4dc8-88dd-f16c7a2e872a/volumes"
Sep 16 13:56:29.071433 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.071393    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b547448-r9hrp"]
Sep 16 13:56:29.072097 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.071599    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65b547448-r9hrp" podUID="97723da7-1f39-49e9-9e45-3a737e8a5b7d" containerName="manager" containerID="cri-o://8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99" gracePeriod=30
Sep 16 13:56:29.325427 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.325356    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:56:29.422554 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.422514    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-metrics-certs\") pod \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") "
Sep 16 13:56:29.422744 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.422581    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skrwb\" (UniqueName: \"kubernetes.io/projected/97723da7-1f39-49e9-9e45-3a737e8a5b7d-kube-api-access-skrwb\") pod \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") "
Sep 16 13:56:29.422744 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.422663    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-webhook-cert\") pod \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\" (UID: \"97723da7-1f39-49e9-9e45-3a737e8a5b7d\") "
Sep 16 13:56:29.424841 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.424808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "97723da7-1f39-49e9-9e45-3a737e8a5b7d" (UID: "97723da7-1f39-49e9-9e45-3a737e8a5b7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:56:29.424977 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.424905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "97723da7-1f39-49e9-9e45-3a737e8a5b7d" (UID: "97723da7-1f39-49e9-9e45-3a737e8a5b7d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:56:29.424977 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.424967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97723da7-1f39-49e9-9e45-3a737e8a5b7d-kube-api-access-skrwb" (OuterVolumeSpecName: "kube-api-access-skrwb") pod "97723da7-1f39-49e9-9e45-3a737e8a5b7d" (UID: "97723da7-1f39-49e9-9e45-3a737e8a5b7d"). InnerVolumeSpecName "kube-api-access-skrwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:56:29.523567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.523529    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skrwb\" (UniqueName: \"kubernetes.io/projected/97723da7-1f39-49e9-9e45-3a737e8a5b7d-kube-api-access-skrwb\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:56:29.523567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.523562    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-webhook-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:56:29.523567 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:29.523571    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97723da7-1f39-49e9-9e45-3a737e8a5b7d-metrics-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Sep 16 13:56:30.169226 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.169191    2577 generic.go:356] "Generic (PLEG): container finished" podID="97723da7-1f39-49e9-9e45-3a737e8a5b7d" containerID="8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99" exitCode=0
Sep 16 13:56:30.169633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.169231    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b547448-r9hrp"
Sep 16 13:56:30.169633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.169263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b547448-r9hrp" event={"ID":"97723da7-1f39-49e9-9e45-3a737e8a5b7d","Type":"ContainerDied","Data":"8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99"}
Sep 16 13:56:30.169633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.169305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b547448-r9hrp" event={"ID":"97723da7-1f39-49e9-9e45-3a737e8a5b7d","Type":"ContainerDied","Data":"5e6f7ab94b08baacfa2021245fe5e556fabd7ac3dd3c7264794bbca1766592a9"}
Sep 16 13:56:30.169633 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.169326    2577 scope.go:117] "RemoveContainer" containerID="8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99"
Sep 16 13:56:30.181569 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.181549    2577 scope.go:117] "RemoveContainer" containerID="8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99"
Sep 16 13:56:30.181939 ip-10-0-134-164 kubenswrapper[2577]: E0916 13:56:30.181915    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99\": container with ID starting with 8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99 not found: ID does not exist" containerID="8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99"
Sep 16 13:56:30.181992 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.181952    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99"} err="failed to get container status \"8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99\": rpc error: code = NotFound desc = could not find container \"8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99\": container with ID starting with 8f2ff86047f4ff978bc0b19471bfc405ebcbf5298726a8252ca1730e8cf22d99 not found: ID does not exist"
Sep 16 13:56:30.193174 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.193149    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65b547448-r9hrp"]
Sep 16 13:56:30.196669 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.196646    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65b547448-r9hrp"]
Sep 16 13:56:30.989732 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.989689    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65b547448-nlq7q"]
Sep 16 13:56:30.990181 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990163    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy"
Sep 16 13:56:30.990181 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990180    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990188    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990194    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990203    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97723da7-1f39-49e9-9e45-3a737e8a5b7d" containerName="manager"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990208    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97723da7-1f39-49e9-9e45-3a737e8a5b7d" containerName="manager"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990219    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcea8893-dba1-4b91-a75b-3f2047180124" containerName="maas-api"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990224    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcea8893-dba1-4b91-a75b-3f2047180124" containerName="maas-api"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990230    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990235    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990243    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9445f919-a104-41fc-ae2e-cdc67a0de187" containerName="maas-api"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990247    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9445f919-a104-41fc-ae2e-cdc67a0de187" containerName="maas-api"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990253    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990257    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990266    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990271    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990276    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990281    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990286    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990292    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990300    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990305    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990310    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990317    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990323    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5ee5e3c-4431-4bba-9e80-922c7fc873a0" containerName="maas-api"
Sep 16 13:56:30.990319 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990328    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5ee5e3c-4431-4bba-9e80-922c7fc873a0" containerName="maas-api"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990401    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990409    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990415    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9445f919-a104-41fc-ae2e-cdc67a0de187" containerName="maas-api"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990421    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990427    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcea8893-dba1-4b91-a75b-3f2047180124" containerName="maas-api"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990433    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990437    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5ee5e3c-4431-4bba-9e80-922c7fc873a0" containerName="maas-api"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990443    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac20dd4-6c19-4382-84b8-bfb05c74ae58" containerName="istio-proxy"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990448    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990453    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d3ba3e5-2a66-4698-802b-99be91ec3ace" containerName="istio-proxy"
Sep 16 13:56:30.991068 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.990463    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97723da7-1f39-49e9-9e45-3a737e8a5b7d" containerName="manager"
Sep 16 13:56:30.994773 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.994757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:30.997929 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.997907    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hqshj\""
Sep 16 13:56:30.998046 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.997911    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 13:56:30.998046 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.997976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 13:56:30.998046 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:30.998014    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 13:56:31.004837 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.004813    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b547448-nlq7q"]
Sep 16 13:56:31.036936 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.036902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ba8df2f-97ce-4883-bc58-c94677e3493f-metrics-certs\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.036936 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.036937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ba8df2f-97ce-4883-bc58-c94677e3493f-webhook-cert\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.037124 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.036974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9gp\" (UniqueName: \"kubernetes.io/projected/6ba8df2f-97ce-4883-bc58-c94677e3493f-kube-api-access-4q9gp\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.110429 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.110393    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97723da7-1f39-49e9-9e45-3a737e8a5b7d" path="/var/lib/kubelet/pods/97723da7-1f39-49e9-9e45-3a737e8a5b7d/volumes"
Sep 16 13:56:31.137627 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.137595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9gp\" (UniqueName: \"kubernetes.io/projected/6ba8df2f-97ce-4883-bc58-c94677e3493f-kube-api-access-4q9gp\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.137785 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.137769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ba8df2f-97ce-4883-bc58-c94677e3493f-metrics-certs\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.137840 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.137801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ba8df2f-97ce-4883-bc58-c94677e3493f-webhook-cert\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.140393 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.140366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ba8df2f-97ce-4883-bc58-c94677e3493f-metrics-certs\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.140490 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.140412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ba8df2f-97ce-4883-bc58-c94677e3493f-webhook-cert\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.146657 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.146638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9gp\" (UniqueName: \"kubernetes.io/projected/6ba8df2f-97ce-4883-bc58-c94677e3493f-kube-api-access-4q9gp\") pod \"maas-controller-65b547448-nlq7q\" (UID: \"6ba8df2f-97ce-4883-bc58-c94677e3493f\") " pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.305585 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.305475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:31.580353 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.580328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b547448-nlq7q"]
Sep 16 13:56:31.581666 ip-10-0-134-164 kubenswrapper[2577]: W0916 13:56:31.581628    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba8df2f_97ce_4883_bc58_c94677e3493f.slice/crio-baa2351511baeb0435d38125cc9cf51d8a1fc48aa651e2ddcdea991906f16bd2 WatchSource:0}: Error finding container baa2351511baeb0435d38125cc9cf51d8a1fc48aa651e2ddcdea991906f16bd2: Status 404 returned error can't find the container with id baa2351511baeb0435d38125cc9cf51d8a1fc48aa651e2ddcdea991906f16bd2
Sep 16 13:56:31.582988 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:31.582972    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:56:32.180916 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:32.180888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b547448-nlq7q" event={"ID":"6ba8df2f-97ce-4883-bc58-c94677e3493f","Type":"ContainerStarted","Data":"baa2351511baeb0435d38125cc9cf51d8a1fc48aa651e2ddcdea991906f16bd2"}
Sep 16 13:56:33.187529 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:33.187497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b547448-nlq7q" event={"ID":"6ba8df2f-97ce-4883-bc58-c94677e3493f","Type":"ContainerStarted","Data":"a9b89b898bec115bac13a552b4400918702fecd41cc42a2211fab1afe6a263d4"}
Sep 16 13:56:33.187980 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:33.187610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 13:56:33.205512 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:33.205461    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65b547448-nlq7q" podStartSLOduration=2.6098273990000003 podStartE2EDuration="3.205444105s" totalImagesPullingTime="595.616706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:33.204238388 +0000 UTC m=+2864.626979972" watchObservedRunningTime="2026-09-16 13:56:33.205444105 +0000 UTC m=+2864.628185687"
Sep 16 13:56:44.195604 ip-10-0-134-164 kubenswrapper[2577]: I0916 13:56:44.195524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65b547448-nlq7q"
Sep 16 14:03:10.333727 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:10.333693    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nfx77_0e58e521-b04f-4495-8451-b7e949b23400/manager/0.log"
Sep 16 14:03:10.451373 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:10.451336    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hggtb_14c58d86-e25f-4d4a-bcac-cd5d5d672adb/manager/0.log"
Sep 16 14:03:10.577238 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:10.577202    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b547448-nlq7q_6ba8df2f-97ce-4883-bc58-c94677e3493f/manager/0.log"
Sep 16 14:03:11.184827 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:11.184795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694ff5d95c-w2j56_6e49a2d2-a9f6-450d-8432-27aff205dde4/manager/0.log"
Sep 16 14:03:11.194102 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:11.194082    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694ff5d95c-w2j56_6e49a2d2-a9f6-450d-8432-27aff205dde4/copy-manifests/0.log"
Sep 16 14:03:11.572424 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:11.572397    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84db685c48-z98rf_c5296ab2-975e-464b-ad17-1428fb2fafde/maas-api/0.log"
Sep 16 14:03:11.698585 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:11.698554    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-48adf5-6696f87674-8n6bq_1a80e599-65c3-4d9a-bdb2-871965459827/maas-api/0.log"
Sep 16 14:03:11.944634 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:11.944607    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w0-a21a40-6c847d7494-dlhvz_428df5de-7c4b-47ce-a02b-ff2e33a01217/maas-api/0.log"
Sep 16 14:03:12.071291 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:12.071261    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w4-f27ea8-85476969c8-5qhk7_291680aa-20e5-4d60-879a-65bfdfa71205/maas-api/0.log"
Sep 16 14:03:13.970022 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:13.969989    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-k6k4h_ae5cac71-0c53-49de-af6b-cb9dba3d1ee3/postgres/0.log"
Sep 16 14:03:14.985412 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:14.985379    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-h9qts_c79c518f-b7b1-4152-bb0c-1bf5338ef94e/manager/0.log"
Sep 16 14:03:15.100344 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:15.100317    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-z4lhk_a575b729-7dd7-4354-880a-d8d5f1d1b933/manager/0.log"
Sep 16 14:03:16.437292 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:16.437263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v5cfk_3523ece8-8647-442a-bef4-8f7af4cf0b01/istio-proxy/1.log"
Sep 16 14:03:18.158846 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:18.158814    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-z62w9_79df9923-d1de-48f3-84f4-f745d3239401/main/0.log"
Sep 16 14:03:18.394001 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:18.393971    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-nk829_c9861481-1ddf-4255-ab62-cee33f299986/main/0.log"
Sep 16 14:03:18.633481 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:18.633449    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4sw9f_605822f9-d69f-4669-964a-28d0ac652bed/main/0.log"
Sep 16 14:03:27.681770 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:27.681705    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vf2vf_5111c756-aa34-4811-a126-504b86fac9d3/global-pull-secret-syncer/0.log"
Sep 16 14:03:27.848396 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:27.848362    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qsn4p_60bd63b3-832a-4c62-a7e9-44fa2a63a540/konnectivity-agent/0.log"
Sep 16 14:03:27.891837 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:27.891807    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-164.ec2.internal_2483e6c65a13c8b4bf2c0714617d8484/haproxy/0.log"
Sep 16 14:03:32.126029 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:32.125993    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-h9qts_c79c518f-b7b1-4152-bb0c-1bf5338ef94e/manager/0.log"
Sep 16 14:03:32.166840 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:32.166811    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-z4lhk_a575b729-7dd7-4354-880a-d8d5f1d1b933/manager/0.log"
Sep 16 14:03:33.970330 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:33.970301    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f630ad5-4918-4957-a9ab-81fcd607aed0/alertmanager/0.log"
Sep 16 14:03:34.000621 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.000593    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f630ad5-4918-4957-a9ab-81fcd607aed0/config-reloader/0.log"
Sep 16 14:03:34.028744 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.028719    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f630ad5-4918-4957-a9ab-81fcd607aed0/kube-rbac-proxy-web/0.log"
Sep 16 14:03:34.058095 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.058073    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f630ad5-4918-4957-a9ab-81fcd607aed0/kube-rbac-proxy/0.log"
Sep 16 14:03:34.083179 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.083157    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f630ad5-4918-4957-a9ab-81fcd607aed0/kube-rbac-proxy-metric/0.log"
Sep 16 14:03:34.131681 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.131655    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f630ad5-4918-4957-a9ab-81fcd607aed0/prom-label-proxy/0.log"
Sep 16 14:03:34.162055 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.162033    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f630ad5-4918-4957-a9ab-81fcd607aed0/init-config-reloader/0.log"
Sep 16 14:03:34.257830 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.257751    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-m77xn_8ff8fbe5-75a8-4cc4-a468-d2c660ce6679/kube-state-metrics/0.log"
Sep 16 14:03:34.286810 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.286757    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-m77xn_8ff8fbe5-75a8-4cc4-a468-d2c660ce6679/kube-rbac-proxy-main/0.log"
Sep 16 14:03:34.313606 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.313569    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-m77xn_8ff8fbe5-75a8-4cc4-a468-d2c660ce6679/kube-rbac-proxy-self/0.log"
Sep 16 14:03:34.347652 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.347624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c856fc477-mws7l_149ded78-e698-4760-a6a8-4346af2a2db3/metrics-server/0.log"
Sep 16 14:03:34.376286 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.376241    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-84mgb_f8894ffc-e25f-45b0-b5ad-d078b2f069cb/monitoring-plugin/0.log"
Sep 16 14:03:34.508997 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.508923    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndhbc_9f39f477-f6c7-4b47-a65d-1810120be64c/node-exporter/0.log"
Sep 16 14:03:34.546186 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.546155    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndhbc_9f39f477-f6c7-4b47-a65d-1810120be64c/kube-rbac-proxy/0.log"
Sep 16 14:03:34.603630 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.603604    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndhbc_9f39f477-f6c7-4b47-a65d-1810120be64c/init-textfile/0.log"
Sep 16 14:03:34.754468 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.754435    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jt9mv_fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657/kube-rbac-proxy-main/0.log"
Sep 16 14:03:34.793033 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.792961    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jt9mv_fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657/kube-rbac-proxy-self/0.log"
Sep 16 14:03:34.825737 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.825713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jt9mv_fc6ddea4-1d1c-4a7c-a1a5-cdb02b122657/openshift-state-metrics/0.log"
Sep 16 14:03:34.883442 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.883412    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_90fdff09-1803-4a13-a5fd-df75392c9a8d/prometheus/0.log"
Sep 16 14:03:34.940034 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.940011    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_90fdff09-1803-4a13-a5fd-df75392c9a8d/config-reloader/0.log"
Sep 16 14:03:34.972661 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:34.972625    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_90fdff09-1803-4a13-a5fd-df75392c9a8d/thanos-sidecar/0.log"
Sep 16 14:03:35.006756 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.006730    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_90fdff09-1803-4a13-a5fd-df75392c9a8d/kube-rbac-proxy-web/0.log"
Sep 16 14:03:35.047340 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.047253    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_90fdff09-1803-4a13-a5fd-df75392c9a8d/kube-rbac-proxy/0.log"
Sep 16 14:03:35.084797 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.084758    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_90fdff09-1803-4a13-a5fd-df75392c9a8d/kube-rbac-proxy-thanos/0.log"
Sep 16 14:03:35.111490 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.111461    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_90fdff09-1803-4a13-a5fd-df75392c9a8d/init-config-reloader/0.log"
Sep 16 14:03:35.146350 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.146324    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-4j2gm_5e441b49-17b3-4e4c-92e2-d263ea8cf8ff/prometheus-operator/0.log"
Sep 16 14:03:35.173744 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.173720    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-4j2gm_5e441b49-17b3-4e4c-92e2-d263ea8cf8ff/kube-rbac-proxy/0.log"
Sep 16 14:03:35.263892 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.263865    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b6f7bd958-4l4hb_a2490c76-8ddf-4190-a31f-1de41d9142e0/telemeter-client/0.log"
Sep 16 14:03:35.301050 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.300931    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b6f7bd958-4l4hb_a2490c76-8ddf-4190-a31f-1de41d9142e0/reload/0.log"
Sep 16 14:03:35.341531 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.341498    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b6f7bd958-4l4hb_a2490c76-8ddf-4190-a31f-1de41d9142e0/kube-rbac-proxy/0.log"
Sep 16 14:03:35.382686 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.382663    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cf57bb6f8-f97v6_52f757c9-aa31-4d10-9fcf-fe2b09a2103c/thanos-query/0.log"
Sep 16 14:03:35.408789 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.408762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cf57bb6f8-f97v6_52f757c9-aa31-4d10-9fcf-fe2b09a2103c/kube-rbac-proxy-web/0.log"
Sep 16 14:03:35.440286 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.440265    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cf57bb6f8-f97v6_52f757c9-aa31-4d10-9fcf-fe2b09a2103c/kube-rbac-proxy/0.log"
Sep 16 14:03:35.466170 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.466144    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cf57bb6f8-f97v6_52f757c9-aa31-4d10-9fcf-fe2b09a2103c/prom-label-proxy/0.log"
Sep 16 14:03:35.494523 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.494476    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cf57bb6f8-f97v6_52f757c9-aa31-4d10-9fcf-fe2b09a2103c/kube-rbac-proxy-rules/0.log"
Sep 16 14:03:35.526521 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:35.526499    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cf57bb6f8-f97v6_52f757c9-aa31-4d10-9fcf-fe2b09a2103c/kube-rbac-proxy-metrics/0.log"
Sep 16 14:03:38.090407 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.090378    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"]
Sep 16 14:03:38.091077 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.091059    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy"
Sep 16 14:03:38.091173 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.091081    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c6d623-32d0-4dc8-88dd-f16c7a2e872a" containerName="istio-proxy"
Sep 16 14:03:38.091236 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.091194    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6d8786-cecf-4069-bd3f-75a654b03198" containerName="istio-proxy"
Sep 16 14:03:38.091236 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.091207    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="53730326-f35c-43cd-9eb6-9dba08bf6d33" containerName="istio-proxy"
Sep 16 14:03:38.094882 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.094842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.098130 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.098108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hrntr\"/\"kube-root-ca.crt\""
Sep 16 14:03:38.098905 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.098885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hrntr\"/\"default-dockercfg-pll6d\""
Sep 16 14:03:38.099010 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.098937    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hrntr\"/\"openshift-service-ca.crt\""
Sep 16 14:03:38.100889 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.100871    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-c4vg9_78bf4632-a381-4760-bec8-3b52200112ac/download-server/0.log"
Sep 16 14:03:38.112140 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.112115    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"]
Sep 16 14:03:38.200060 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.200007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-proc\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.200260 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.200075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-sys\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.200260 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.200133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-podres\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.200260 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.200165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-lib-modules\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.200443 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.200427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ljs\" (UniqueName: \"kubernetes.io/projected/4e3237e8-e808-4520-b91a-55fdfd6eaf25-kube-api-access-v6ljs\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301175 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-proc\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301366 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-sys\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301366 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-podres\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301366 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-lib-modules\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301366 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-proc\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301366 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-sys\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301536 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-podres\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301536 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e3237e8-e808-4520-b91a-55fdfd6eaf25-lib-modules\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.301536 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.301389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ljs\" (UniqueName: \"kubernetes.io/projected/4e3237e8-e808-4520-b91a-55fdfd6eaf25-kube-api-access-v6ljs\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.310411 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.310384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ljs\" (UniqueName: \"kubernetes.io/projected/4e3237e8-e808-4520-b91a-55fdfd6eaf25-kube-api-access-v6ljs\") pod \"perf-node-gather-daemonset-8xslh\" (UID: \"4e3237e8-e808-4520-b91a-55fdfd6eaf25\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.406558 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.406474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:38.682439 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:38.682414    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"]
Sep 16 14:03:38.684428 ip-10-0-134-164 kubenswrapper[2577]: W0916 14:03:38.684401    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e3237e8_e808_4520_b91a_55fdfd6eaf25.slice/crio-30267bb732528347f15274456d91b4ba70863cf1b0f15ee3b0feb01ae3c3e6f7 WatchSource:0}: Error finding container 30267bb732528347f15274456d91b4ba70863cf1b0f15ee3b0feb01ae3c3e6f7: Status 404 returned error can't find the container with id 30267bb732528347f15274456d91b4ba70863cf1b0f15ee3b0feb01ae3c3e6f7
Sep 16 14:03:39.403169 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:39.403136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh" event={"ID":"4e3237e8-e808-4520-b91a-55fdfd6eaf25","Type":"ContainerStarted","Data":"c608cf74ee2f40a25ae93c638f863b7636435479a943f72eabb6da40f865cc69"}
Sep 16 14:03:39.403608 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:39.403234    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:39.403608 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:39.403263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh" event={"ID":"4e3237e8-e808-4520-b91a-55fdfd6eaf25","Type":"ContainerStarted","Data":"30267bb732528347f15274456d91b4ba70863cf1b0f15ee3b0feb01ae3c3e6f7"}
Sep 16 14:03:39.428433 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:39.428386    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh" podStartSLOduration=1.428373314 podStartE2EDuration="1.428373314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:39.426781027 +0000 UTC m=+3290.849522606" watchObservedRunningTime="2026-09-16 14:03:39.428373314 +0000 UTC m=+3290.851114893"
Sep 16 14:03:40.005598 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:40.005565    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r5lhr_38ca1309-d752-41bd-ab24-91703d4ee35e/dns/0.log"
Sep 16 14:03:40.025078 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:40.025052    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r5lhr_38ca1309-d752-41bd-ab24-91703d4ee35e/kube-rbac-proxy/0.log"
Sep 16 14:03:40.048759 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:40.048718    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hqrcn_3f870356-a81b-4032-8134-6a9419c26220/dns-node-resolver/0.log"
Sep 16 14:03:40.810273 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:40.810242    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rm9l6_c4195754-df4b-4f6e-90b3-cecdf8415aeb/node-ca/0.log"
Sep 16 14:03:42.369760 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:42.369724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v5cfk_3523ece8-8647-442a-bef4-8f7af4cf0b01/istio-proxy/0.log"
Sep 16 14:03:42.371090 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:42.371072    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v5cfk_3523ece8-8647-442a-bef4-8f7af4cf0b01/istio-proxy/1.log"
Sep 16 14:03:43.649709 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:43.649676    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mgxfc_456f9723-c1e7-46f3-9746-aeccf33c3271/serve-healthcheck-canary/0.log"
Sep 16 14:03:44.457725 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:44.457698    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqf4x_b2242fad-8a9b-43bc-a5bb-1ea4508bae38/kube-rbac-proxy/0.log"
Sep 16 14:03:44.479929 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:44.479907    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqf4x_b2242fad-8a9b-43bc-a5bb-1ea4508bae38/exporter/0.log"
Sep 16 14:03:44.499675 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:44.499653    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqf4x_b2242fad-8a9b-43bc-a5bb-1ea4508bae38/extractor/0.log"
Sep 16 14:03:45.418567 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:45.418537    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-8xslh"
Sep 16 14:03:46.756675 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:46.756641    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nfx77_0e58e521-b04f-4495-8451-b7e949b23400/manager/0.log"
Sep 16 14:03:46.918320 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:46.918223    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hggtb_14c58d86-e25f-4d4a-bcac-cd5d5d672adb/manager/0.log"
Sep 16 14:03:46.993213 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:46.993177    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b547448-nlq7q_6ba8df2f-97ce-4883-bc58-c94677e3493f/manager/0.log"
Sep 16 14:03:47.383745 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:47.383711    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694ff5d95c-w2j56_6e49a2d2-a9f6-450d-8432-27aff205dde4/manager/0.log"
Sep 16 14:03:47.469388 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:47.469370    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694ff5d95c-w2j56_6e49a2d2-a9f6-450d-8432-27aff205dde4/copy-manifests/0.log"
Sep 16 14:03:48.290928 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:48.290899    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v5cfk_3523ece8-8647-442a-bef4-8f7af4cf0b01/istio-proxy/0.log"
Sep 16 14:03:48.293938 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:48.293915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v5cfk_3523ece8-8647-442a-bef4-8f7af4cf0b01/istio-proxy/1.log"
Sep 16 14:03:48.792429 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:48.792404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b97dbbc45-85rrr_1da303e1-7f13-4600-bf42-18ab63b5d09b/manager/0.log"
Sep 16 14:03:50.353484 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:50.353451    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v5cfk_3523ece8-8647-442a-bef4-8f7af4cf0b01/istio-proxy/0.log"
Sep 16 14:03:50.355063 ip-10-0-134-164 kubenswrapper[2577]: I0916 14:03:50.355041    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v5cfk_3523ece8-8647-442a-bef4-8f7af4cf0b01/istio-proxy/1.log"